This Notebook contains my work for analyzing a 100km transect in the Southern Ocean. The goal is to identify Lee waves from the measurements and then estimate the energy fluxes and transport driven by these lee waves. For some of the more standard plots such as transect contours, I also experiment with my Ocean Toolbox package.
Playing around with the integration limits seems to not change the energy or frequency too much but the horizontal wavelengths do seem sensitive to it.
If I want to do the energy flux and ray tracing, I think I will pick a couple select packets and isolate them to get more concrete wavenumbers to use in ray tracing.
The peaks in internal energy visually correlate well with the velocity anamoly graphs where theres packets of similar anomalies. However, I am not sure why the wave numbers are so big (>100km).
# Load Data and relevant modules - this is common to both wayss
%matplotlib inline
import numpy as np
import scipy.signal as sig
import matplotlib.patches as mpatches
import matplotlib.pyplot as plt
import data_load
import gsw
import oceans as oc
import pandas as pd
import internal_waves_calculations as iwc
import warnings
import cmocean
import matplotlib.colors as colors
import ray_tracing as rt
# Probably Shouldn't do this but they annoy me
warnings.simplefilter("ignore")
pd.options.display.max_rows = 3000
pd.options.display.max_columns = 22
plt.rcParams.update({'font.size':14})
# load data and cut off bottom (its all nans)
ladcp, ctd = data_load.load_data()
strain = np.genfromtxt('strain.csv', delimiter=',')
wl_max = 1000
wl_min = 400
ctd_bin_size = 1024
ladcp_bin_size = 1024
nfft = 1024
U, V, p_ladcp = oc.loadLADCP(ladcp)
S, T, p_ctd, lat, lon = oc.loadCTD(ctd)
rho = gsw.density.rho(S, T, p_ctd)
maxDepth = 4000
idx_ladcp = p_ladcp[:, -1] <= maxDepth
idx_ctd = p_ctd[:, -1] <= maxDepth
S = S[idx_ctd,:]
T = T[idx_ctd,:]
rho = rho[idx_ctd,:]
p_ctd = p_ctd[idx_ctd, :]
U = U[idx_ladcp, :]
V = V[idx_ladcp, :]
p_ladcp = p_ladcp[idx_ladcp, :]
# Bin CTD data
ctd_bins = oc.binData(S, p_ctd[:, 0], ctd_bin_size)
# Bin Ladcp Data
ladcp_bins = oc.binData(U, p_ladcp[:, 0], ladcp_bin_size)
# Depth and lat/long grids (For plots)
depths = np.vstack([np.nanmean(p_ctd[binIn]) for binIn in ctd_bins])
dist = gsw.distance(lon, lat)
dist = np.cumsum(dist)/1000
dist = np.append(0,dist)
Isopycnal displacements using neutral density from CSIRO matlab function
N2_ref, N2, strain, p_mid, rho_bar = oc.adiabatic_level(S,
T,
p_ctd,
lat,
pressure_range=400, # Difference window for leveling
order=1, # order of fit
axis=0,
)
# Neutral Densities
rho_neutral = np.genfromtxt('neutral_densities.csv', delimiter=',')
rho_n = rho_neutral[idx_ctd,:]
# Poly fit to neutral density to get reference profiles
ref = []
for cast in rho_n.T:
fitrev = oc.vert_polyFit2(cast, p_ctd[:, 0], 100, deg=2)
ref.append(fitrev)
ref = np.vstack(ref).T
ref = np.nanmean(ref, axis=1)
ref2 = np.tile(ref,(21,1)).T
eta = oc.isopycnal_displacements(rho_n, ref2, p_ctd, lat)
This section uses spectral analysis to estimate kinetic and potential energies lee waves. Using a minimum and maximum vertical wavelength as integration limits, I estimate the energetics of target wavelengths. These limits were determined qualitatively by estimative vertical size of features in profiles (how big are the wiggles).
The wave components are estimated through calcuation of the internal wave energy components (Kinetic and Potential). To do this, each profile is seperated into mean and wave-induced perturbations $(X = X + X')$. In order to do this a sliding vertical polynomial is fit to each profile and the resultant profile is subtracted out, leaving the perturbation profile.
The resultant velocity perturbation profiles (u and v) are binned into 1024 meter, half overlapping bins. The Power Spectral Density is calculated along each being and integrated between a target wavelength band. This band is chosen by examining the perturbation profiles and identifying coherent wave features. The final values are input into the kinetic energy equation. $\langle \rangle$ denotes integrated power spectral density.
$$ KE = \frac{1}{2}\big[ \langle u'^{2} \rangle + \langle v'^{2} \rangle \big] $$
A similar process is followed for estimating potential energy. However, 2 methods were implemented here, with varying results. The first was to estimate the isopycnal displacement $\eta$ from density perturbations as follows:
$$\eta = \frac{\rho - \rho_{ref}}{\frac{\Delta \rho_{ref}}{\Delta z}}$$
$\rho_{ref}$ is estimated as a transect mean density profile. See Potential Energy Notebook for this clalculation worked out Once $\eta$ is obtained, the same processes for calculating Power Spectral Density used in kinetic energy calculations are used here with the potential energy equation:
$$ PE = \frac{1}{2}N^2\langle \eta'^{2} \rangle $$
The energy components are combined for the total internal energy $(\frac{J}{m^s})$:
$$ E = \rho_{0}(KE + PE) $$
Using the two energy components, the internal wave frequencies are estimated by:
$$ \omega_{0} = f\sqrt{\frac{KE + PE}{KE - PE}}$$
Several other derivations are used and compared with similar results. An issue with this method is that when kinetic and potential energies are similar, error in energy density calculations may cause kinetic energy to be slightly less than potential, resulting in a null value from the square root operation. We now have the intrinsic frequency as well as the vertical waveumnber $m$ which is estimated as the mean of integration limits. From here, $k_{h}$, the horizontal wave number is calculated from:
$$ k_{h} = m\sqrt{\frac{f^2 - \omega^2}{\omega^2 - N^2}} $$ (Meyer 2016)
# set integration limits by vertical wavelength
wl_max = 1000
wl_min = 500
# Calculate KE spectrums (m2/s2)
z_ladcp = -1*gsw.z_from_p(p_ladcp, lat)
KE, KE_grid, KE_psd, Uprime, Vprime, ke_peaks = iwc.KE_UV(U, V, z_ladcp, ladcp_bins,
wl_min, wl_max, lc=wl_min-50,
nfft=1024, detrend='constant')
# Calculate PE spectrum using eta from above (m2/s2)
z_ctd = -1*gsw.z_from_p(p_ctd, lat)
PE, PE_grid, eta_psd, N2mean, pe_peaks = iwc.PE(N2, z_ctd, eta,
wl_min, wl_max,
ctd_bins, nfft=1024,
detrend='constant')
m_plot = np.array([(1)/wl_max,
(1)/wl_max, (1)/wl_min,
(1)/wl_min])
plt.figure(figsize=[12,6])
plt.loglog(KE_grid, KE_psd.T, linewidth=.6, c='b', marker='x', alpha=.05)
plt.loglog(KE_grid, np.nanmean(KE_psd, axis=0).T, lw=2, c='b')
ylims = plt.gca().get_ylim()
ylim1 = np.array([ylims[0], ylims[1]])
plt.plot(m_plot[2:], ylim1, lw=1,
c='k', alpha=.5,
linestyle='dotted')
plt.plot(m_plot[:2], ylim1, lw=1,
c='k', alpha=.5,
linestyle='dotted')
plt.ylim(ylims)
plt.ylabel('Kinetic Energy Density')
plt.xlabel('Vertical Wavenumber')
plt.gca().grid(True, which="both", color='k', linestyle='dotted', linewidth=.2)
plt.loglog(PE_grid, .5*np.nanmean(N2)*eta_psd.T,
lw=.6, c='r', marker='x', alpha=.05)
plt.loglog(PE_grid, .5*np.nanmean(N2)*np.nanmean(eta_psd, axis=0).T,
lw=2, c='r')
plt.plot(m_plot[2:], ylim1, lw=1,
c='k', alpha=.5,
linestyle='dotted')
plt.plot(m_plot[:2], ylim1, lw=1,
c='k', alpha=.5,
linestyle='dotted')
plt.ylim((1e-8, 1e2))
# plt.gca().grid(True, which="both", color='k', linestyle='dotted', linewidth=.2)
plt.ylabel(r'Potential Energy Density $m^2s^2$')
plt.xlabel('Vertical Wavenumber')
plt.title('Kinetic And Potential Energy Spectrum')
# plt.xlim(.0005, .01)
ax = plt.gca()
plt.text(4e-4, 1e-4,r'$\lambda$ = {} m'.format(wl_max))
plt.text(2.8e-3, 1e-7,r'$\lambda$ = {} m'.format(wl_min))
potential = mpatches.Patch(color='red', label='Potential Energy')
kinetic = mpatches.Patch(color='b', label='Kinetic Energy')
plt.legend(handles=[kinetic, potential])
In order to properly run a ray tracing model, the horizontal wavenumber $k_h$ must be decomposed into its two components, $k$ and $l$. The horizontal azimuth, $\theta$, is the angle between the $k_h$ vector and the x-axis. using the relationships: $$ tan(2\theta) = 2\mathbb{R} \bigg [\frac{u'^{*}v'}{u'u^{*} - v'v^{*}} \bigg ]$$
$$ k = (kh)cos(\theta) $$ $$ l = (kh)sin(\theta) $$
where $*$ represents the complex conjugate and $u'$ and $v'$ represent the Fourier transform of velocity anomalies. Fourier transforms are done along the same bins used in the energy calculations. NOT SO SURE ABOUT HOW RIGHT THIS IS
# ENERGY AND WAVE COMPONENT PLOTS
Etotal = 1027*(KE + PE) # Multiply by density to get Joules
# wave components
f = np.nanmean(gsw.f(lat))
# version 2 omega calculation - where did this come from?
omega = f*np.sqrt(((KE+PE)/(KE-PE))) # Waterman et al. 2012 (Ithink)
# m = (2*np.pi)/800
m = (2*np.pi)*np.nanmean(ke_peaks, axis=1)
m = np.reshape(m, KE.shape, order='F')
kh = m*np.sqrt(((f**2 - omega**2)/(omega**2 - N2mean))) # Waterman et al. 2012
kh2 = (m/np.sqrt(N2mean))*(np.sqrt(omega**2 - f**2)) # Where (meyer i think?)
lambdaH = 1e-3*(2*np.pi)/kh
lambdaH2 = 1e-3*(2*np.pi)/kh2
# version 2 omega calculation
Rw = KE/PE # Unsure what to do with this just yet.
# Wave vector decomposition
k = []
l = []
theta = []
dz = 8
for i in ladcp_bins:
theta.append(iwc.horizontal_azimuth(Uprime[i,:], Vprime[i,:], dz,\
wl_min=wl_min,
wl_max=wl_max,
nfft=1024))
theta = np.vstack(theta)
k = kh*np.cos(theta)
l = kh*np.sin(theta)
plt.figure(figsize=[15,18])
plt.subplot(611)
plt.pcolormesh(dist.flatten(), depths.flatten(), (Etotal), cmap=cmocean.cm.matter)
c1 = plt.colorbar(extend='max')
c1.set_label(r'$J/m^3$')
plt.gca().invert_yaxis()
plt.xlabel('Distance along transect')
plt.ylabel('Depth')
plt.title(r'Total Internal Wave Energy $\frac{J}{m^{3}}$ ')
mask = lambdaH > 150
norm=colors.LogNorm(vmin=lambdaH.min(), vmax=lambdaH.max())
# lambdaH[mask] = np.nan
plt.subplot(612)
plt.pcolormesh(dist.flatten(), depths.flatten(), (lambdaH), cmap=cmocean.cm.curl, norm=None)
c2 = plt.colorbar(extend='max')
c2.set_label(r'km')
plt.gca().invert_yaxis()
plt.xlabel('Distance along transect')
plt.title('Horizontal Wavelength Vector Magnitude ')
plt.subplot(613)
plt.pcolormesh(dist.flatten(), p_ladcp.flatten(), np.abs(Uprime), cmap=cmocean.cm.delta)
c3 = plt.colorbar(extend='max')
c3.set_label(r'$m/s$')
plt.xlabel('Distance along transect')
plt.ylabel('Depth')
plt.title(r"$|u'|$")
plt.ylim(0,3000)
plt.gca().invert_yaxis()
plt.subplot(614)
plt.pcolormesh(dist.flatten(), p_ladcp.flatten(), np.abs(Vprime), cmap=cmocean.cm.delta)
c3 = plt.colorbar(extend='max')
c3.set_label(r'$m/s$')
plt.xlabel('Distance along transect')
plt.ylabel('Depth')
plt.title(r"$|v'|$")
plt.ylim(0,3000)
plt.gca().invert_yaxis()
plt.tight_layout()
plt.subplot(615)
plt.pcolormesh(dist.flatten(), p_ladcp.flatten(), U, cmap=cmocean.cm.delta, shading='flat')
c3 = plt.colorbar(extend='max')
c3.set_label(r'$m/s$')
plt.xlabel('Distance along transect')
plt.ylabel('Depth')
plt.title(r"$U $")
plt.ylim(0,3000)
plt.gca().invert_yaxis()
plt.tight_layout()
plt.subplot(616)
plt.pcolormesh(dist.flatten(), p_ladcp.flatten(), V, cmap=cmocean.cm.delta, shading='flat')
c3 = plt.colorbar(extend='max')
c3.set_label(r'$m/s$')
plt.xlabel('Distance along transect')
plt.ylabel('Depth')
plt.title(r"$V $")
plt.ylim(0,3000)
plt.gca().invert_yaxis()
plt.tight_layout()
# # Experiment with peaks and see what happens KINDA USELESS CODE BUT INTERESTING TO LOOK AT
# # split u and v peaks (they come together as output of the kinetic energy function)
# u_peaks = ke_peaks[:,0]
# v_peaks = ke_peaks[:,1]
# # Reshape the arrays to match the binned grids
# # fortran order reshapes it the right way ( I was to lazy to do it earlier)
# u_peaks = np.reshape(u_peaks, KE.shape, order='F')
# oc.display(u_peaks, index=depths.flatten())
# mask = 1/u_peaks > 1700
# u_peaks[mask] = np.nan
# plt.figure(figsize=[20,20])
# ax = plt.gca()
# ax.matshow(u_peaks)
# for (i, j), z in np.ndenumerate(1/u_peaks):
# ax.text(j, i, '{:0f}'.format(z), ha='center', va='center',
# bbox=dict(boxstyle='round', facecolor='white', edgecolor='0.3'))
# # plt.pcolormesh(dist.flatten(), depths.flatten(), 1/u_peaks, cmap=cmocean.cm.matter)
# # c1 = plt.colorbar(extend='max')
# # c1.set_label(r'$J/m^3$')
# # plt.gca().invert_yaxis()
# # plt.xlabel('Distance along transect')
# # plt.ylabel('Depth')
# # plt.title(r'Total Internal Wave Energy $\frac{J}{m^{3}}$ ')
The peaks in internal energy visually correlate well with the velocity anamoly graphs where theres packets of similar anomalies. However, I am not sure why the wave numbers are so big (>100km).
table = oc.display(omega, index=depths.flatten())
table.style.set_caption('Horizontal Wavelength V1')
fig = plt.figure(figsize=[10,6])
plt.subplot(121)
plt.hist(lambdaH.flatten(), bins=15, range=(np.nanmin(lambdaH), np.nanmax(lambdaH)))
plt.title(r'$\lambda_h$')
plt.xlabel(r'Horizontal Wavelength Magnitude (km)')
plt.tight_layout()
plt.subplot(122)
plt.hist(omega.flatten(), bins=20, range=(np.nanmin(omega), np.nanmax(omega)))
plt.title(r'$\omega_0$')
plt.xlabel(r'Frequency ($s^{-1}$)')
plt.tight_layout()
ax = plt.gca()
ax.ticklabel_format(style='sci', scilimits=(1,1), axis='x')
table = oc.display(k, index=depths.flatten())
table.style.set_caption(r'Horizontal Wavenumber $k$')
table = oc.display(l, index=depths.flatten())
table.style.set_caption(r'Horizontal Wavenumber $l$')
table = oc.display(m, index=depths.flatten())
table.style.set_caption(r'Horizontal Wavenumber $m$')
Ray tracing equations are following Olbers 1981 as follows:
Horizontal Group Speeds: $$ C_{gx} = k\frac{N^2 - \omega^2}{\omega(kh^2 + m^2)} $$ $$ C_{gy} = l\frac{N^2 - \omega^2}{\omega(kh^2 + m^2)} $$ Vertical Group Speed: $$ C_{gz} = m\frac{\omega^2 - f^2}{\omega(kh^2 + m^2)} $$ Change in vertical wavenumber: $$ \frac{dm}{dt} \approx \frac{dm}{dz} = -\big (k\frac{du}{dz} + l\frac{dv}{dz} \big) $$
Group speeds are multiplied by the specified time step to get x, y, z movements. horizontal wavenumber components are assumed constant (for now). Vertical wavenumber changes with depth (velocity and buoyancy profiles are constant in x,y, and time.
l1 = -0.000139543
k1 = 0.000324448
m1 = -0.00976433
z0 = 1500
w0 = -0.000132755
wave1 = rt.wave(k=k1, l=l1, m=m1, w0=w0, z0=z0)
# check that the properties are loaded correctly by using the properties attribute
wave1.properties()
duration = 48
tstep = 10
status = 6 # intervals to give run status
wave1.back3d(duration=duration, tstep=tstep, status=status, print_run_report=True)
fig1 = plt.figure(figsize=[10,10])
wave1.x_m_plot(fig=fig1, cmap='Reds', line_colorbar=True)
plt.title('Ray Tracing - Run Duration: {} hours'.format(duration))
depths2 = np.tile(depths, (21,1))
seafloor = 4000
status = False
waves = []
for i in range(KE.flatten().shape[0]):
if np.sum(np.isfinite([k.flatten()[i], l.flatten()[i], m.flatten()[i], omega.flatten()[i]])) == 4:
waves.append(rt.wave(k=k.flatten()[i],
l=l.flatten()[i], m=-m.flatten()[i],
w0=omega.flatten()[i], z0=depths2.flatten()[i]))
duration = 48
tstep = 10
status = 6
seafloor = 4000
for wave in waves:
wave.back3d(duration=duration, tstep=tstep,
status=status, seafloor=seafloor,
updates=False, print_run_report=False)
# Plot ray tracing (Cell is split to make plot changes without having to re run the model)
plt.figure(figsize=[10,8])
for wave in waves:
plt.plot(wave.x_ray.flatten(), wave.z_ray.flatten(), label=r'$\omega$ = {}'.format(wave.w0_init))
plt.gca().invert_yaxis()
plt.title('Frequency Experiment')
plt.ylabel('Depth (meters)')
plt.xlabel('x - Distance (km) ')
Thorpe scales are used to estimate the local dissipation rate, which will be compared to the estimated dissipation rate of the obeserved internal waves. For each bin, the density profile is resorted into monotonic order and the root mean square of the vertical displacements is calculated as the Thorpe Scale. This length scale is related empirically to the dissipation rate following the work of [@Meter et al. 2015].


GRODVI
Section: User Commands (1)Updated: 16 September 2002
Index Return to Main Contents
NAME
grodvi - convert groff output to TeX dvi formatSYNOPSIS
grodvi [ -dv ] [ -wn ] [ -Fdir ] [ files... ]It is possible to have whitespace between a command line option and its parameter.
DESCRIPTION
grodvi is a driver for groff that produces TeX dvi format. Normally it should be run by groff -Tdvi. This will run troff -Tdvi; it will also input the macros /usr/share/groff/<IP_ADDRESS>/tmac/dvi.tmac; if the input is being preprocessed with eqn it will also input /usr/share/groff/<IP_ADDRESS>/font/devdvi/eqnchar.The dvi file generated by grodvi can be printed by any correctly-written dvi driver. The troff drawing primitives are implemented using the tpic version~2 specials. If the driver does not support these, the [rs]D commands will not produce any output.
There is an additional drawing command available:
- [rs]D'R dh dv'
- Draw a rule (solid black rectangle), with one corner at the current position, and the diagonally opposite corner at the current position +(dh,dv). Afterwards the current position will be at the opposite corner. This produces a rule in the dvi file and so can be printed even with a driver that does not support the tpic specials unlike the other [rs]D commands.
The groff command [rs]X'anything' is translated into the same command in the dvi file as would be produced by [rs]special{anything} in TeX; anything may not contain a newline.
For inclusion of EPS image files, grodvi loads pspic.tmac automatically, providing the PSPIC macro. Please check grops (1) for a detailed description of this macro.
Font files for grodvi can be created from tfm files using tfmtodit(1). The font description file should contain the following additional commands: u+2n
- internalname name The name of the tfm file (without the .tfm extension) is name.
- checksum n
- The checksum in the tfm file is n.
- designsize n
- The designsize in the tfm file is n.
These are automatically generated by tfmtodit.
The default color for [rs]m and [rs]M is black. Currently, the drawing color for [rs]D commands is always black, and fill color values are translated to gray.
In troff the [rs]N escape sequence can be used to access characters by their position in the corresponding tfm file; all characters in the tfm file can be accessed this way.
OPTIONS
- -d
- Do not use tpic specials to implement drawing commands. Horizontal and vertical lines will be implemented by rules. Other drawing commands will be ignored.
- -v
- Print the version number.
- -wn
- Set the default line thickness to n thousandths of an em. If this option isn't specified, the line thickness defaults to 0.04~em.
- -Fdir
- Prepend directory dir/devname to the search path for font and device description files; name is the name of the device, usually dvi.
USAGE
There are styles called R, I, B, and BI mounted at font positions 1 to 4. The fonts are grouped into families T and H having members in each of these styles:-
- TR
- CM Roman (cmr10)
- TI
- CM Text Italic (cmti10)
- TB
- CM Bold Extended Roman (cmbx10)
- TBI
- CM Bold Extended Text Italic (cmbxti10)
- HR
- CM Sans Serif (cmss10)
- HI
- CM Slanted Sans Serif (cmssi10)
- HB
- CM Sans Serif Bold Extended (cmssbx10)
- HBI
- CM Slanted Sans Serif Bold Extended (cmssbxo10)
There are also the following fonts which are not members of a family:
-
- CW
- CM Typewriter Text (cmtt10)
- CWI
- CM Italic Typewriter Text (cmitt10)
Special fonts are MI (cmmi10), S (cmsy10), EX (cmex10), and, perhaps surprisingly, TR, TI, and CW, due to the different font encodings of text fonts. For italic fonts, CWI is used instead of CW.
Finally, the symbol fonts of the American Mathematical Society are available as special fonts SA (msam10) and SB (msbm10). These two fonts are not mounted by default.
Using the option -mec (loading the file ec.tmac) EC and TC fonts are used. The design of the EC family is very similar to that of the CM fonts; additionally, they give a much better coverage of groff symbols. Note that ec.tmac must be called before any language-specific files; it doesn't take care of hcode values.
FILES
- /usr/share/groff/<IP_ADDRESS>/font/devdvi/DESC
- Device description file.
- /usr/share/groff/<IP_ADDRESS>/font/devdvi/F
- Font description file for font F.
- /usr/share/groff/<IP_ADDRESS>/tmac/dvi.tmac
- Macros for use with grodvi.
- /usr/share/groff/<IP_ADDRESS>/tmac/ec.tmac
- Macros to switch to EC fonts.
BUGS
Dvi files produced by grodvi use a different resolution (57816 units per inch) to those produced by TeX. Incorrectly written drivers which assume the resolution used by TeX, rather than using the resolution specified in the dvi file will not work with grodvi.When using the -d option with boxed tables, vertical and horizontal lines can sometimes protrude by one pixel. This is a consequence of the way TeX requires that the heights and widths of rules be rounded.
SEE ALSO
tfmtodit(1), groff(1), troff(1), groff_out(5), groff_font(5), groff_char(7)
Index
This document was created by man2html, using the manual pages.
Time: 05:29:04 GMT, December 24, 2015


Todoist Productivity Stats
Reset API Token
GitHub Project
Hourly Completed Tasks Breakdown
Select Graph Options
Range:
to
Project:
ALL
Load Graph
Daily Completed Task Breakdown
Select Graph Options
Range:
to
Project:
ALL
Load Graph


This is a POC application to demonstrate the Push notifications service using socket.io + redis + postgres.
The notification service has been developed in a plugin style to be able to embed into cross-framework nodejs applications,
atleast that's the idea as it's yet to be put to test in this regard.
< Click Me to clear notifications below Last Notification Recieved @ {{getLastUpdated}}


Malians rally to support army as 20 bodies found in village
by AFPThousands of people took to the streets of Mali's capital Bamako on Friday to support the West African country's army, which has suffered heavy losses as it struggles to quell a jihadist revolt.
The rally came as a grim discovery in the fragile Sahel nation's conflict-wracked centre illustrated the size of the challenge the army faces.
"Twenty bodies, some abandoned in wells, were found" by soldiers in the village of Peh in the central Mopti region on Wednesday, the government said.
The soldiers were sent to the village after reports of an attack by "unidentified individuals dressed in hunter's clothes," the government said in a statement Friday.
The group "abandoned stolen herds and retreated" when the soldiers arrived, the statement added, without saying how the 20 people died.
A rising jihadist insurgency that spread from the north to the centre of Mali has inflamed tensions between herders and sedentary farmers. Hundreds of lives have been lost in tit-for-tat assaults.
In Bamako, protest organisers said around 15,000 people marched in support of the army, while the police put the number at "several thousand".
Demonstrators held up signs saying "No embezzlement of our funds" and "Long live FAMA (Malian armed forces)".
Opposition leader Soumaila Cisse said the government had betrayed the army.
"Our soldiers are ill-equipped. We should be fighting for our military to be up to the job," he said, blaming corruption.
Former minister and opposition figure Mohamed Ali Bathily said "the military and ministers who are behind this corruption must be judged".
In the last month around 100 Malian soldiers have been killed in two of the deadliest attacks in the country's history.
The army has been battling to contain the Islamist insurgency despite help from France, African neighbours and the United Nations.


Sainsbury's launches Giving Store to let you donate food to those in need
If, like us, you love the joy of browsing shelves but simply can’t justify buying anything else for yourself, rejoice.

Oh, and it’ll let you do some good as well.
How, you may ask? Well, it’s all down to one simple concept: buying food and other goods for those who actually need the items, rather than for yourself.
The Giving Store, located in Covent Garden, will let people browse aisles and take their pick of what they fancy picking up, which they can then donate at the tills to people in need.
Not only do you get to enjoy the act of shopping, but you can also feel the joy of giving. Which is what Christmas is all about, right?
Advertisement Advertisement
Sainsbury’s is urging parents to bring along their children to the store to teach them about the importance of thinking of others during the festive period.
Don’t worry, the experience won’t be like your average grocery shop. The Giving Store has special characters taking kids through the process of creating a Christmas dinner and discovering a snow-filled forest, up to the interactive donation containers where kids and parents can drop off their items.
To get in, adults will need to pay £5, which gets donated directly to a local food bank. Children go free.
Judith Batchelar, the director of Sainsbury’s Brand, said: ‘If every person doing their Christmas shopping this December donated one extra item – be that a can of soup or a roll-on deodorant – over 50 million products could be donated to those in need this festive season.
‘We want to do what we can to encourage shoppers to donate food and toys and do so in a way that teaches children the importance of helping others, particularly around the festive season’.
Nice idea, we think.
If you fancy popping along, the Sainsbury’s Giving Store is open from Thursday 5 December to Sunday 8 December, from 4pm to 8pm on Thursday then 11am to 8pm on all the other days.
It’s right in the heart of central London, at 11 Short Gardens, WC2H 9AT, so you can easily tag on a visit when you go to look at the Christmas lights.
Advertisement Advertisement


Event is emitted when a user's presence or information changes.
event: "PRESENCE_UPDATE"
NewPresence: Presence
The updated Presence instance
OldPresence?: Presence
The previous presence object, this is not present if no previous presence exists for the user
This event is emitted when a user's presence or information such as name or avatar are updated.
| Connecting to Discord |
|


About Us
A Sugar House-based python web developer meetup. We are interested in promoting python development, but also welcome wisdom found in other programming languages. Avid and aspiring programmers alike are welcome to join us.
We alternate our meeting between the first Wednesday and the first Thursday of the month at Zaniac Sugar House (the old Sugar House coffee location; the big purple building).
For current news follow us using Meetup.com


Investing with easy
Choose a pack and invest in MoMo-Coin with ease with means of payment accessible by all.
Download the application now and get easier access to MobileMoney Coin.


Ploror688.github.io
Becas de Investigación en Monash University - Australia Las becas para programas de Investigación están. We are not an institution so please contact the relevant institution directly for course information. Andrea Fraser – Galerie Nagel Draxler Curated by professional editors, The Conversation offers informed commentary and debate on the issues affecting our world. Plus a Plain English guide to the latest. Student Profiles — IITB-Monash Research Academy. INTI International College Subang - An Introduction . Located in the heart of the Klang Valley, where education, business and modern infrastructure converge SCCI - Sherman Centre for Culture and Ideas. Our Teachers — The Craft Sessions. International opportunities - Monash University.
SCCI is a vibrant platform for the exchange of challenging ideas on fashion and architecture. Girls can do anything. At Branksome Hall Asia, girls take center stage as they learn
Campus Malaysia - Malaysia University Course Application. Master of arts creative writing monash. International experience is invaluable. That's why Monash have secured global relationships to provide international education research opportunities. The University of Adelaide offers the highest number of specialist options of any Bachelor of Arts (BA) in South Australia. Our BA gives
Our mission is to engage and connect international students all over the world, with a focus on Asia and Middle East regions, to higher education opportunities.
Upper Canada College - Toronto Private Day Boarding School.
Jeffrey Tucker is Editorial Director at the American Institute for Economic Research. He is founder of Liberty.me, Distinguished Honorary Member of Mises Brazil. United Kingdom. The Master of Letters degree is a postgraduate degree awarded by a few select British and Irish universities, predominantly within the ancient English. John Monash Scholars - General Sir John Monash Foundation. Monash University is one of Australia’s leading universities and ranks among the world’s top 100. We help change lives through research and education. Learn.
ANDREA FRASER geboren / born 1965 in Billings, Montana AUSBILDUNG / EDUCATION 1985-86 New York University. 1984-85 Whitney Museum of American. Considering Upper Canada College in Toronto? OurKids.net is the trusted source on the best private schools in Ontario. George TS has done his Master’s in Advanced Manufacturing Engineering from NITK Surathkal and has last worked as a Research Associate working on the development. The Hopkins-Nanjing Center for Chinese and American Studies (HNC) opened in 1986 as a one-of-a-kind graduate center for international studies in China. Becas Empleos - BECAS en la Argentina y el exterior. The Craft Sessions is a weekend retreat of creative workshops, delicious food, quiet moments and inspiring people. Workshop themes include knitting, quilting. Speakers ALS Friedman Conference. Recipients of the John Monash Scholarships are recognised as John Monash Scholars. Our University Scholarships Postgraduate Scholarships are awarded to….
Arts + Culture – Views Research – The Conversation. Private university - Wikipedia. INTI International College Subang Campus Malaysia.
Monash University - one of the top universities in Australia. Bachelor of Arts Degree Finder.


Water, Earth, Fire, Air. You will not find those things here.
In case you couldn't tell, I'm a nerd.


Add this code to your CSS file and set the class colorful to the element you want to have the background.
HC


Element's name on the Lua API. You usually set it to <NAMESPACE>_PT_<ELEMENTNAME>, in order to tell the executable what your element ID variable will be called inside Lua.
An example value could be:
Identifier = "DEFAULT_PT_BCOL";
In Lua scripting, you can set it with:
local element_id = elements.allocate("DEFAULT", "BCOL")


Packaging Datasets with Mantra¶
With Mantra it’s easy to package data for deep learning. In these docs, we are going to see how we make a data package, and how we process it using the powerful Dataset
class.
💾 Make a Dataset¶
Go to the root of your project. To make a dataset use the makedata
command. We can make an empty dataset as follows:
$ mantra makedata first_dataset
Or if we already have a tar.gz file with some data, we can reference it as follows:
$ mantra makedata first_dataset --tar-path tar_path_here
Our new data folder will be located at myproject/data/first_dataset. Inside:
raw/
__init__.py
data.py
README.md
data.py
contains the coreDataset
class that is used to process your dataraw/
contains the tar.gz file with the raw dataREADME.md
is where you can describe the model (useful for sharing the model with others)
If we don’t need flat files, but want to import data through an API, we can use the no-tar flag:
$ mantra makedata first_dataset --no-tar
We now need to extract the input and output vectors X, y that are used to train models ...
✨ Magic Data Templates¶
Many datasets are standardised, such as a folder of images or a csv file with columns of features and labels. Mantra provides magic templates so you don’t have the write the entire class yourself.
💁 Images¶
If we have a tar file that contains a folder of images, we can use the images
template:
$ mantra makedata celeba --template 'images' --tar-path celebA.tar.gz --image-dim 128 128
Above we are using the images template. This will create an ImageDataset
class using the tar file provided. We can also specify additional default options for the template:
Parameter Type Example Description –image-dim list 64 64 Desired image dimension (height, output) –normalize bool (flag) –normalize Whether to normalize the images for training
Once we have executed the command, we can open the data.py
file:
import numpy as np
from mantraml.data import Dataset, cachedata
from mantraml.data import ImageDataset
class MyImageDataset(ImageDataset):
data_name = 'My Image Dataset'
data_tags = ['example', 'new', 'images']
files = ['celebA.tar.gz']
image_dataset = 'celebA.tar.gz' # referring to the file that contains the images
# additional default data
has_labels = False
image_dim = (128, 128)
normalized = True
@cachedata
def y(self):
# return your labels here as an np.ndarray
# if no labels, e.g. generative models, then you can remove this method
return
We can see that we are inheriting from ImageDataset
. We can also see our input dimensions have entered as a default argument. We can use sample
to eyeball the data:
from data.celeba.data import MyImageDataset
dataset = MyImageDataset(name='celeba')
dataset.sample()
So the advantage of using a template is that we didn’t have to write any code. We could, if we wish though, write on top of these templates for some further customisation if we needed it.
📙 Tables¶
If we have a flat csv file, we can use the tabular
template to configure it:
$ mantraml makedata table_data --template 'tabular' --tar-path mydata.tar.gz
$ --file-name 'my_flat_file.csv' --target 'target_column'
$ --features 'feature_1' 'feature_2'
This will create an TabularDataset
class. We can also specify additional options for the template.
Parameter Type Example Description –file-name str ‘my_flat_file.csv’ The name of the flat file inside the tar –target str ‘target_column’ The column name to extract as the target –features list ‘feature_1’ ‘feature_2’ The columns to extract as the features –target-index int 0 The column index of the target –features-index list 1 2 The column indices to extract as features
The index options are there if we want to refer to the table by indices rather than column names; if we just want to use column names then we can ignore these options.
Once we have executed the command, we can open the data.py
file:
import numpy as np
from mantraml.data import TabularDataset
class MyTabularDataset(TabularDataset):
data_name = 'Example Table Data'
files = ['mydata.tar.gz']
data_file = 'my_flat_file.csv'
data_tags = ['tabular']
has_labels = True
target = 'target_column'
features = ['feature_1', 'feature_2']
We can see that we are inheriting from TabularDataset
. We can also see our feature and target options are now default argument options. This dataset is now Mantra ready. If we want to alter features from the command line:
$ mantraml train my_model --dataset table_data --features feature_1 feature_2 feature_3
📊 Custom Data Processing¶
If the magic templates aren’t useful, you can write your own data processing logic. Open up the data.py
file:
import numpy as np
from mantraml.data import Dataset, cachedata
class MyImageDataset(Dataset):
data_name = 'My Image Dataset'
data_tags = ['example', 'new', 'images']
files = ['myfiles.tar.gz']
has_labels = False
@cachedata
def X(self):
# return your features here as an np.ndarray
return
@cachedata
def y(self):
# return your labels here as an np.ndarray
return
Simply write your logic for extracting X, y in the above. Your dependency data in files
will be extracted to a path at self.self.extracted_data_path
. So if you are extracting data from these files, just open the files from this directory and do what you want with them.
You might be wondering what the @cachedata
decorator does. It does two things. First it is a property based decorator so you can access the data at MyImageDataset().X
and MyImageDataset().y
respectively. Secondly, it caches the data to RAM upon the first call so the processing logic doesn’t have to be run twice. If you don’t want the caching, then just replace this decorator with @property
.
For more more Mantra dataset examples, check out the Mantra examples repository.
🖼 Visualizing Your Data Projects¶
Load up the UI and click on a model:
$ mantra ui
In order to customise how the UI looks for your dataset you can add metadata to your dataset classes:
class PremierLeagueData(MantraModel):
# The Name of the Dataset
data_name = 'Premier League Data'
# The Dataset Image
data_image = "default.jpeg"
# Link to a Notebook
data_notebook = 'notebook.ipynb'
# Tags for the Model
data_tags = ['football', 'epl']
🤖 Accessing Datasets in Models¶
When you define a model you pass in a data
and task
parameter:
def __init__(self, data=None, task=None, **kwargs):
self.data = data
self.task = task
If you don’t have a task, then you have no training/test split, and you can simply access the data at self.data.X
and self.data.y
.
If you have a task then you can train your model on the training set explicitly at self.task.X_train
and self.task.y_train
.


Flex Hero is a simple CSS library to exploit flexbox layouts.
This is NOT a grid layout system or a complete CSS framework. On contrary, Flex Hero can be used with any other CSS framework, like Bootstrap.
Flex Hero is available on Bower and NPM:
npm install flex-hero
bower install flex-hero


Node, gRPC, and Postgres
gRPC is an open source remote procedure call (RPC) framework developed by Google. It’s built on top of HTTP/2, and it uses Protocol Buffers as the underlying data serialization format.
This tutorial looks at how to implement an API with Node, gRPC, and Postgres.
Contents
Learning Objectives
- Explain what gRPC is and how it compares to REST
- Define the service definition and payload message structure using a Protocol Buffer
- Describe the four types of methods–unary, client streaming, server streaming, and bidirectional streaming
- Create a gRPC server and client in Node
- Invoke the gRPC server from the client via the gRPC stubs
- Describe the static and dynamic approaches to Protocol Buffers in Node
gRPC
As mentioned, gRPC is an RPC framework that leverages HTTP/2 and Protocol Buffers, making it a fast, secure, and reliable communication protocol for microservices.
New to gRPC? Start with the official What is gRPC? guide.
So, how does gRPC compare to REST?
We’ll use the most common implementation of REST for this comparison: An API that uses the HTTP verbs–GET, POST, PUT, DELETE–to accept and return JSON.
- Speed: HTTP/2 supports bidirectional streams (and flow control), so once a connection is established the client or server can push and pull data. Check out this demo of HTTP/1.1 vs. HTTP/2 for a performance comparison. Also, you can read all about the features in HTTP/2 here.
- Messages (instead of resources and verbs): With gRPC you are no longer constrained to a limited set of methods. You can create your own methods, like
getCustomerList
oremailCustomerOrderReceipt
, that can be leveraged by simply calling the method. And, from a developer’s perspective, calling a gRPC method looks and feels just like calling any other native local method–it’s just making a network call beneath the scenes. - Protocol Buffers: gRPC uses Protocol Buffers (strongly typed, binary-based) rather than JSON (loosely typed, text-based), which is much more efficient and introduces type safety. The service definitions themselves are defined declaratively and are used to generate client libraries. Further, Protocol Buffers, when compared to JSON, can decrease the size of the payloads being transmitted, which, at scale, can reduce bandwidth cost.
For more on gRPC vs REST, review the REST vs. gRPC: Battle of the APIs blog post.
Project Setup
Create a new directory to hold the project:
$ mkdir node-grpc-crud
$ cd node-grpc-crud
If you don’t want to code along, you can find the final code on GitHub.
Then, add the following files and folders:
├── client
│ ├── app.js
│ └── package.json
├── protos
└── server
├── index.js
└── package.json
Finally, spin up PostgreSQL on port 5432 and create a new database called grpc_products
.
gRPC Service
Throughout this tutorial, you’ll be building a product API backed by gRPC that handles the basic CRUD functionality.
Let’s start by implementing a gRPC Service, which is defined by the methods it exposes along with it’s associated parameters and return message types.
Add a new file to the “protos” folder called product.proto to hold the gRPC service definition and message type definitions:
syntax = "proto3";
package product;
// service definition
service ProductService {
rpc listProducts(Empty) returns (ProductList) {}
rpc readProduct(ProductId) returns (Product) {}
rpc createProduct(newProduct) returns (result) {}
rpc updateProduct(Product) returns (result) {}
rpc deleteProduct(ProductId) returns (result) {}
}
// message type definitions
message Empty {}
message ProductList {
repeated Product products = 1;
}
message ProductId {
int32 id = 1;
}
message Product {
int32 id = 1;
string name = 2;
string price = 3;
}
message newProduct {
string name = 1;
string price = 2;
}
message result {
string status = 1;
}
Using VSCode? Check out the proto3 vscode extension.
First, we used the proto3 version of the Protocol Buffer language:
syntax = "proto3";
Then, we defined a package specifier:
package product;
The package specifier is optional but it’s generally a good idea to use to avoid name clashes. Review the JavaScript Generated Code reference guide for more info.
Next, we defined five RPC methods that align to the basic CRUD operations:
| CRUD | gRPC method |
|---|---|
| Read | listProducts |
| Create | createProduct |
| Read | readProduct |
| Update | updateProduct |
| Delete | deleteProduct |
There are four types of methods in gRPC-land:
| Type | Description | Example |
|---|---|---|
| Unary | client sends a single request and gets a single response | rpc getData(req) returns (rsp) {} |
| Server streaming | client sends a single request and gets back a stream | rpc getData(req) returns (stream rsp) {} |
| Client streaming | client sends stream and gets back a single response | rpc getData(stream req) returns (rsp) {} |
| Bidirectional streaming | both the client and server send and receive streams | rpc getData(stream req) returns (stream rsp) {} |
Our example uses the Unary approach.
The first method, listProducts
, takes an Empty
message and returns a ProductList
message, which has a repeated Product
field called products
:
message ProductList {
repeated Product products = 1;
}
The repeated keyword is used to define an array of objects.
We also created an Empty
message as the empty stub for empty requests and responses:
message Empty {}
The next method, readProduct
, takes a ProductId
and returns a Product
:
message Product {
int32 id = 1;
string name = 2;
string price = 3;
}
This has three scalar value fields, each with a unique numbered tag:
- id (int32)
- name (string)
- price (string)
We created a ProductId
as well with a single int32 field:
message ProductId {
int32 id = 1;
}
Next, createProduct
takes a newProduct
and returns a result
, which will be just a status message of “success” or “failure”. Take note of the newProduct
message and the updateProduct
and deleteProduct
methods on your own.
With the service defined, you can now generate interfaces in a number of different languages.
Server
Moving on, let’s create the gRPC server that will be used to serve up the remote procedure calls.
Setup
Start by updating the package.json file in the “server” folder:
{
"name": "node-grpc-server",
"dependencies": {
"@grpc/proto-loader": "^0.4.0",
"google-protobuf": "^3.6.1",
"grpc": "^1.18.0",
"knex": "^0.16.3",
"pg": "^7.8.0"
},
"scripts": {
"start": "node index.js"
}
}
Take note of the dependencies:
- @grpc/proto-loader: loads .proto files
- google-protobuf: JavaScript version of the Protocol Buffers runtime library
- grpc: Node gRPC Library
- knex: SQL query builder for Node
- pg: PostgreSQL client for Node
Before installing, you will need to install protoc
, the Protobuf Compiler. If you’re on a Mac, it’s easiest to install with Homebrew:
$ brew install protobuf
Otherwise, you can manually download and install it from here.
Now you can install the NPM dependencies:
$ cd server
$ npm install
Next, add a knexfile.js file to the “server” folder:, which is used for configuring knex for different environments–e.g., local, development, production:
const path = require('path');
module.exports = {
development: {
client: 'postgresql',
connection: {
host: '<IP_ADDRESS>',
user: '',
password: '',
port: '5432',
database: 'grpc_products',
},
pool: {
min: 2,
max: 10,
},
migrations: {
directory: path.join(__dirname, 'db', 'migrations'),
},
seeds: {
directory: path.join(__dirname, 'db', 'seeds'),
},
},
};
Add the appropriate username and password. Then, create the a “db” directory along with the “migrations” and “seeds” directories. Your project structure should now look like:
├── client
│ ├── app.js
│ └── package.json
├── protos
│ └── product.proto
└── server
├── db
│ ├── migrations
│ └── seeds
├── index.js
├── knexfile.js
├── package-lock.json
└── package.json
Create a new migration file:
$ ./node_modules/.bin/knex migrate:make products
Then, add the following code to the migration stub in “server/db/migrations”:
exports.up = function (knex, Promise) {
return knex.schema.createTable('products', function (table) {
table.increments();
table.string('name').notNullable();
table.string('price').notNullable();
});
};
exports.down = function (knex, Promise) {
return knex.schema.dropTable('products');
};
Apply the migration:
$ ./node_modules/.bin/knex migrate:latest
Create a new seed file:
$ ./node_modules/.bin/knex seed:make products
Add the code:
exports.seed = function (knex, Promise) {
// Deletes ALL existing entries
return knex('products').del()
.then(function () {
// Inserts seed entries
return knex('products').insert([
{ name: 'pencil', price: '1.99' },
{ name: 'pen', price: '2.99' },
]);
});
};
Apply the seed:
$ ./node_modules/.bin/knex seed:run
Finally, wire up the basic server in index.js:
// requirements
const path = require('path');
const protoLoader = require('@grpc/proto-loader');
const grpc = require('grpc');
// knex
const environment = process.env.ENVIRONMENT || 'development';
const config = require('./knexfile.js')[environment];
const knex = require('knex')(config);
// grpc service definition
const productProtoPath = path.join(__dirname, '..', 'protos', 'product.proto');
const productProtoDefinition = protoLoader.loadSync(productProtoPath);
const productPackageDefinition = grpc.loadPackageDefinition(productProtoDefinition).product;
/*
Using an older version of gRPC?
(1) You won't need the @grpc/proto-loader package
(2) const productPackageDefinition = grpc.load(productProtoPath).product;
*/
// knex queries
function listProducts(call, callback) {}
function readProduct(call, callback) {}
function createProduct(call, callback) {}
function updateProduct(call, callback) {}
function deleteProduct(call, callback) {}
// main
function main() {
const server = new grpc.Server();
// gRPC service
server.addService(productPackageDefinition.ProductService.service, {
listProducts: listProducts,
readProduct: readProduct,
createProduct: createProduct,
updateProduct: updateProduct,
deleteProduct: deleteProduct,
});
// gRPC server
server.bind('localhost:50051', grpc.ServerCredentials.createInsecure());
server.start();
console.log('gRPC server running at http://<IP_ADDRESS>:50051');
}
main();
Here, we load the gRPC service definition, create a new gRPC server, add the service to the server, and then run the server on http://localhost:50051.
Take note of the addService
method:
server.addService(productPackageDefinition.ProductService.service, {
listProducts: listProducts,
readProduct: readProduct,
createProduct: createProduct,
updateProduct: updateProduct,
deleteProduct: deleteProduct,
});
This method adds the gRPC service, defined in the product.proto, to the server. It takes the service definition–e.g., ProductService
–along with an object which maps the method names from the gRPC service to the method implementations defined above.
Dynamic vs Static Code Generation
There are two ways of working with Protocol Buffers in Node:
- Dynamically: with dynamic code generation, the Protocol Buffer is loaded and parsed at run time with Protobuf.js
- Statically: with the static approach, the Protocol Buffer is pre-processed into JavaScript
We used the dynamic approach above. The dynamic approach is quite a bit simpler to implement, but it differs from the workflow of other gRPC-supported languages, since they require static code generation.
Want to use the static approach?
First, install grpc-tools globally:
$ npm install -g grpc-tools
Then, run the following from the project root:
$ protoc -I=. ./protos/product.proto \
--js_out=import_style=commonjs,binary:./server \
--grpc_out=./server \
--plugin=protoc-gen-grpc=`which grpc_tools_node_protoc_plugin`
The generated code should now be in the “server/protos” directory:
- product_grpc_pb.js
- product_pb.js
You’ll then import the generated code into your index.js file and use them directly rather than loading the service definition.
Sanity Check
Try running the server at this point:
$ npm start
You should see:
gRPC server running at http://<IP_ADDRESS>:50051
Let’s wire up the gRPC client before adding the knex query functions.
├── client
│ ├── app.js
│ └── package.json
├── protos
│ └── product.proto
└── server
├── db
│ ├── migrations
│ │ └──<PHONE_NUMBER>4532_products.js
│ └── seeds
│ └── products.js
├── index.js
├── knexfile.js
├── package-lock.json
└── package.json
Client
Like before, update the package.json file the “client” directory:
{
"name": "node-grpc-client",
"dependencies": {
"@grpc/proto-loader": "^0.4.0",
"body-parser": "^1.18.3",
"express": "^4.16.4",
"google-protobuf": "^3.6.1",
"grpc": "^1.18.0"
},
"scripts": {
"start": "node app.js"
}
}
From the “client” directory, install the dependencies:
$ npm install
Update app.js:
// requirements
const express = require('express');
const bodyParser = require('body-parser');
const productRoutes = require('./routes/productRoutes');
// express
const app = express();
app.use(bodyParser.urlencoded({ extended: false }));
app.use(bodyParser.json());
// routes
app.use('/api', productRoutes);
// run server
app.listen(3000, () => {
console.log('Server listing on port 3000');
});
Here, we instantiate a new Express server, define our routes, and then run the server. Add a “routes” folder along with the Express router in a productRoutes.js file:
// requirements
const express = require('express');
const grpcRoutes = require('./grpcRoutes');
// new router
const router = express.Router();
// routes
router.get('/products', grpcRoutes.listProducts);
router.get('/products/:id', grpcRoutes.readProduct);
router.post('/products', grpcRoutes.createProduct);
router.put('/products/:id', grpcRoutes.updateProduct);
router.delete('/products/:id', grpcRoutes.deleteProduct);
module.exports = router;
Finally, add the boilerplate for the gRPC client in client/routes/grpcRoutes.js:
// requirements
const path = require('path');
const protoLoader = require('@grpc/proto-loader');
const grpc = require('grpc');
// gRPC client
const productProtoPath = path.join(__dirname, '..', '..', 'protos', 'product.proto');
const productProtoDefinition = protoLoader.loadSync(productProtoPath);
const productPackageDefinition = grpc.loadPackageDefinition(productProtoDefinition).product;
const client = new productPackageDefinition.ProductService(
'localhost:50051', grpc.credentials.createInsecure());
/*
Using an older version of gRPC?
(1) You won't need the @grpc/proto-loader package
(2) const productPackageDefinition = grpc.load(productProtoPath).product;
(3) const client = new productPackageDefinition.ProductService(
'localhost:50051', grpc.credentials.createInsecure());
*/
// handlers
const listProducts = (req, res) => {};
const readProduct = (req, res) => {};
const createProduct = (req, res) => {};
const updateProduct = (req, res) => {};
const deleteProduct = (req, res) => {};
module.exports = {
listProducts,
readProduct,
createProduct,
updateProduct,
deleteProduct,
};
Test it out:
$ npm start
You should see:
Server listing on port 3000
Kill the server before moving on.
├── client
│ ├── app.js
│ ├── package-lock.json
│ ├── package.json
│ └── routes
│ ├── grpcRoutes.js
│ └── productRoutes.js
├── protos
│ └── product.proto
└── server
├── db
│ ├── migrations
│ │ └──<PHONE_NUMBER>4532_products.js
│ └── seeds
│ └── products.js
├── index.js
├── knexfile.js
├── package-lock.json
└── package.json
With that, we’re now ready to tie everything together.
CRUD
We’ll use the following workflow for wiring up the gRPC stubs:
- Add the appropriate knex query to the server (server/index.js)
- Add the associated handler to the client (client/routes/grpcRoutes.js)
- Test it via cURL
listProducts
Server:
function listProducts(call, callback) {
/*
Using 'grpc.load'? Send back an array: 'callback(null, { data });'
*/
knex('products')
.then((data) => { callback(null, { products: data }); });
}
Client:
const listProducts = (req, res) => {
/*
gRPC method for reference:
listProducts(Empty) returns (ProductList)
*/
client.listProducts({}, (err, result) => {
res.json(result);
});
};
To test, run the client in one terminal window and the server in another. Then, run the following command:
$ curl http://<IP_ADDRESS>:3000/api/products
You should see the products:
{
"products": [
{
"id": 1,
"name": "pencil",
"price": "1.99"
},
{
"id": 2,
"name": "pen",
"price": "2.99"
}
]
}
readProduct
Server:
function readProduct(call, callback) {
knex('products')
.where({ id: parseInt(call.request.id) })
.then((data) => {
if (data.length) {
callback(null, data[0]);
} else {
callback('That product does not exist');
}
});
}
Client:
const readProduct = (req, res) => {
const payload = { id: parseInt(req.params.id) };
/*
gRPC method for reference:
readProduct(ProductId) returns (Product)
*/
client.readProduct(payload, (err, result) => {
if (err) {
res.json('That product does not exist.');
} else {
res.json(result);
}
});
};
Test success:
$ curl http://<IP_ADDRESS>:3000/api/products/1
{
"id": 1,
"name": "pencil",
"price": "1.99"
}
Test failure:
$ curl http://<IP_ADDRESS>:3000/api/products/99999
"That product does not exist."
createProduct
Server:
function createProduct(call, callback) {
knex('products')
.insert({
name: call.request.name,
price: call.request.price,
})
.then(() => { callback(null, { status: 'success' }); });
}
Client:
const createProduct = (req, res) => {
const payload = { name: req.body.name, price: req.body.price };
/*
gRPC method for reference:
createProduct(newProduct) returns (result)
*/
client.createProduct(payload, (err, result) => {
res.json(result);
});
};
Test:
$ curl -X POST -d '{"name":"lamp","price":"29.99"}' \
-H "Content-Type: application/json" http://<IP_ADDRESS>:3000/api/products
{
"status": "success"
}
$ curl http://<IP_ADDRESS>:3000/api/products
{
"products": [
{
"id": 1,
"name": "pencil",
"price": "1.99"
},
{
"id": 2,
"name": "pen",
"price": "2.99"
},
{
"id": 3,
"name": "lamp",
"price": "29.99"
}
]
}
updateProduct
Server:
function updateProduct(call, callback) {
knex('products')
.where({ id: parseInt(call.request.id) })
.update({
name: call.request.name,
price: call.request.price,
})
.returning()
.then((data) => {
if (data) {
callback(null, { status: 'success' });
} else {
callback('That product does not exist');
}
});
}
Client:
const updateProduct = (req, res) => {
const payload = { id: parseInt(req.params.id), name: req.body.name, price: req.body.price };
/*
gRPC method for reference:
updateProduct(Product) returns (result)
*/
client.updateProduct(payload, (err, result) => {
if (err) {
res.json('That product does not exist.');
} else {
res.json(result);
}
});
};
Test:
$ curl -X PUT -d '{"name":"lamp","price":"49.99"}' \
-H "Content-Type: application/json" http://<IP_ADDRESS>:3000/api/products/3
{
"status": "success"
}
$ curl http://<IP_ADDRESS>:3000/api/products/3
{
"id": 3,
"name": "lamp",
"price": "49.99"
}
deleteProduct
Server:
function deleteProduct(call, callback) {
knex('products')
.where({ id: parseInt(call.request.id) })
.delete()
.returning()
.then((data) => {
if (data) {
callback(null, { status: 'success' });
} else {
callback('That product does not exist');
}
});
}
Client:
const deleteProduct = (req, res) => {
const payload = { id: parseInt(req.params.id) };
/*
gRPC method for reference:
deleteProduct(ProductId) returns (result)
*/
client.deleteProduct(payload, (err, result) => {
if (err) {
res.json('That product does not exist.');
} else {
res.json(result);
}
});
};
Test:
$ curl -X DELETE http://<IP_ADDRESS>:3000/api/products/3
{
"status": "success"
}
$ curl http://<IP_ADDRESS>:3000/api/products
{
"products": [
{
"id": 1,
"name": "pencil",
"price": "1.99"
},
{
"id": 2,
"name": "pen",
"price": "2.99"
}
]
}
What if the product doesn’t exist?
$ curl -X DELETE http://<IP_ADDRESS>:3000/api/products/9999
"That product does not exist."
Conclusion
gRPC provides a declarative, strongly typed mechanism for defining an API.
Workflow:
- Define the API (service definition and message structure) using a Protocol Buffer
- Implement the server using the generated interface
- Add the message stubs to the client
Resources:


layoutMenu with contentStack single view
Content Pane 1
panel 1 client
Content Pane 2
panel 2
layout index page
Content Pane 3
panel 3
Content Pane 4
panel 4


The Research Revealed newsletter brings you behind the scenes so you can make the most of the vast research operations support network at BU. Please submit your newsletter ideas, and visit the Research Support website for more information.
Sponsored Programs and the Office of Research are now accepting nominations for this year’s Excellence in Research Administration Award, which honors two staff members — one in central administration and the other at a school or center — for exceptional service to the University's research mission.
Two new Affinity Research Collaboratives — one on tobacco regulatory science and the other on fibrosis in pathology — have been established to promote team science.
The BU School of Medicine is considering expanding the Metabolic Phenotyping and IVIS® Imaging cores and is conducting a survey to assess faculty members’ needs for mouse vascular or cardiovascular phenotyping.
The Office of Research is seeking faculty input on what types of topics would be most helpful for University researchers to explore in the Strategic Communications Series.
Pre-proposals for the Johnson & Johnson Innovation Lung Cancer Center pilot projects are due on Sept. 7.
Nominations for the BU Innovator of the Year Award are due on Sept. 14.
Applications for the Social Innovation on Drug Resistance (SIDR) Postdoctoral Program are due on Sept. 28.
Fallou Ngom, a CAS professor of anthropology and the director of the African Studies Center at the Pardee School of Global Studies, won a $250k collaborative research grant from the National Endowment for the Humanities for a project that will research and prepare for online and print publication texts written in West African languages that use Arabic script.
Federal funding for BU’s African Studies Center was also recently renewed by the U.S. Department of Education, which awarded more than $550k in continued support for both the center and its Foreign Language Area Studies fellowship program.
The Nanotechnology Innovation Center has awarded seed funding to projects led by Chen Yang (ENG), Xue Han (ENG), Xin Zhang (ENG), and Stephan Anderson (MED).
The Pardee Center for the Study of the Longer Range Future published a conference report stemming from its Symposium on Global Health & the Social Sciences.
The Initiative on Cities released Leading the American City, a five-part video history of mayoral leadership in the United States.
The Danielsen Institute has been awarded the Templeton Flourishing Grant for clinical and training research related to mental health care.
Shereen Shermak has been named the new executive director of the Hariri Institute-affiliated Boston Women’s Workforce Council.
New per diem animal rates have been announced.
The federal Institute of Education Sciences has made available various materials, including transcripts, Powerpoint presentations, and on-demand video webinars, to help researchers apply for funding opportunities.
Post Award Financial Operations has created a new onboarding pack for departments and updated its training video series with an entry on salary accrual for faculty on nine-month appointments.
Registrations for the 2018 a2ru conference are now open.
Get an in-depth look at research administration, support, and resources at BU. Learn how to make the most of services in sponsored programs, bring a product to market, stay compliant, and interact with industry, government, and the media. Faculty and staff from any discipline are welcome to attend!
Location: Kilachand Center for Integrated Life Sciences & Engineering
A Call to Arms: Advancing Women’s Health Research in the Military Symposium
Research How-To | The Current Changing Landscape at NSF with Focus on Geosciences
Research on Tap | Medicine in the Molecular Era: Single Cell Sequencing
Strategic Communications Series | Distill Your Message
Research on Tap | Current Research on Autism at BU: From Cells to Society
Research on Tap | Understanding and Forecasting Change in Our Natural World
Research on Tap | Re-Engineering Life: Tissue Engineering in Health and the Environment
22nd Annual BU Photonics Center Symposium on Frontiers of Cardiac Tissue Engineering


There was an error submitting please try again
Artist: Marco Amedani
Software Developer: Dan Fredell, Keun Kindred
The lights are too hot. They will be cooled off at
Take a screenshot of the next page and share with us on
Instagram and Facebook.
5
Take a screenshot of the next page and share with us on Instagram and Facebook.
Your light will be displayed at
5
Artist: Marco Amedani
Software Developer: Dan Fredell, Keun Kindred


 It is important to remember
that a Job is connected to one Group only. To create a Job, you should select the
appropriate Unit Type, Job Type, Service Window and Group to get started. See the
field help for more information.
Dependency
Jobs can have a dependency, which means the Jobs will have to wait for the Job
dependency to complete. Because of this you can make structures of Jobs that will run in
sequence. This is useful if you need to make a sequence like this: Disable something-Configure-Enable something.
Another use case might be Reboot-Configure.
Stop rules
You can also set stop rules for the Job, and this is made easy with a GUI that pops up
when you hit the plus sign on the right side of the input field for the stop rules. These
settings will help you to define when the Job should stop.
An example to this is 10 Unconfirmed failed Unit Jobs out of 1000. When you hit the Add
failure rule it will be converted to u10/1000 in the text field. An unconfirmed failure
is when the device never returns to the server after a change has been issued. This is
considered a very dangerous situation - it might indicate that the change cause the device
to loose connection. The other type of failure is Confirmed failure which means that
the device rejected the change with an error message. The idea is that a Job can
stop automatically when a certain condition arises.
Job states
To start the Job, hit the START button.
Note that you cannot change any parameters set if the Job has been started, so make
sure you have everything set correct before you start the Job. You can at any time pause
the job, and then restart it.
Repeat Jobs
Certain jobs may be necessary to run more than once. To do so set the repeat-counter to
a positive number (number of repeats) and set the repeat interval to a number of seconds.
This interval will override (if necessary) the default spread interval decided by
the service window. However, the job will still work within the service window.
Job parameters
The job parameters tell the job what the change is all about. Here you specify what actually
should be configured on the device. If you choose Job type to be SOFTWARE, then you will
only be allowed to set DesiredSoftwareVersion. Other rules apply for other job types.


Anuj Yadav
Top Scorer
100
Total Users
220
Highest Score
Liked Us
PLAY AGAIN
Country Finder
Howdy Amigo!
250
Your Score
87%
Ahead Of
Easy
Difficulty
Top Scores
Brad Pitt
Intermediate
250
Tom Hanks
Easy
230
Al Pacino
Player of the Week
210
Robert De Niro
Player of the Day
180
Recent Players


Your emailed cards appear in ...
Tip: Don't share this email address. Anyone who has it can add cards as you. When composing emails, the card title goes in the subject and the card description in the body. More email formatting tips.
Your emailed cards appear in ...
Tip: Don't share this email address. Anyone who has it can add cards as you. When composing emails, the card title goes in the subject and the card description in the body. More email formatting tips.


If you're having issues with the site, find any bugs, or just have questions, send us a message and we'll be in touch.
Auburn, WA 98042, USA
1-800-273-8255
contact@foodie.com


Send us a message about whatever you like, ask a question or tell us about your next job.
Use the contact form to get in touch, and if you’re looking to hire us for a project, fill in as much detail about the project as you can.
Contact Information:
Address:
No.28 - 63739 street lorem ipsum City, Country
Phone:
+ 1<PHONE_NUMBER>
Email:
support@yoursite.com


| PERL5200DELTA(1) | Perl Programmers Reference Guide | PERL5200DELTA(1) |
If you are upgrading from an earlier release such as 5.16.0, first read perl5180delta, which describes differences between 5.16.0 and 5.18.0.
See "Signatures" in perlsub for details.
For example, "sub foo($$){}" could be rewritten as "sub foo : prototype($$){}".
Whitespace has long been allowed inside subroutine prototypes, so "sub( $ $ )" is equivalent to "sub($$)", but until now it was stripped when the subroutine was parsed. Hence, whitespace was not allowed in prototypes set by "Scalar::Util::set_prototype". Now it is permitted, and the parser no longer strips whitespace. This means "prototype &mysub" returns the original prototype, whitespace and all.
This meant that the quality of perl's random numbers would vary from platform to platform, from the 15 bits of rand() on Windows to 48-bits on POSIX platforms such as Linux with drand48().
Perl now uses its own internal drand48() implementation on all platforms. This does not make perl's "rand" cryptographically secure. [perl #115928]
$sref->$*; # same as ${ $sref } # interpolates $aref->@*; # same as @{ $aref } # interpolates $href->%*; # same as %{ $href } $cref->&*; # same as &{ $cref } $gref->**; # same as *{ $gref } $aref->$#*; # same as $#{ $aref } $gref->*{ $slot }; # same as *{ $gref }{ $slot } $aref->@[ ... ]; # same as @$aref[ ... ] # interpolates $href->@{ ... }; # same as @$href{ ... } # interpolates $aref->%[ ... ]; # same as %$aref[ ... ] $href->%{ ... }; # same as %$href{ ... }
Those marked as interpolating only interpolate if the associated "postderef_qq" feature is also enabled. This feature is experimental and will trigger "experimental::postderef"-category warnings when used, unless they are suppressed.
For more information, consult the Postfix Dereference Syntax section of perlref.
The regular expression engine now supports strings longer than 2**31 characters. [perl #112790, #116907]
The functions PerlIO_get_bufsiz, PerlIO_get_cnt, PerlIO_set_cnt and PerlIO_set_ptrcnt now have SSize_t, rather than int, return values and parameters.
You can still use -p for its extra behaviour. [perl #116190]
A literal "{" after "\B" or "\b" is now fatal.
These were deprecated in perl v5.14.0.
Details are in "Beyond Unicode code points" in perlunicode.
If you have tests that depend on the exact output of Data::Dumper, they may fail.
To avoid this problem in your code, test against the data structure from evaluating the dumped structure, instead of the dump itself.
This worked well until VC++ 2010 and later, which introduced new E* constants with values > 100 into errno.h, including some being (re)defined by perl to WSAE* values. That caused problems when linking XS code against other libraries which used the original definitions of errno.h constants.
To avoid this incompatibility, perl now maps WSAE* error codes to E* values where possible, and assigns those values to $!. The E* constants exported by Errno and POSIX are updated to match so that testing $! against them, wherever previously possible, will continue to work as expected, and all E* constants found in errno.h are now exported from those modules with their original errno.h values.
In order to avoid breakage in existing Perl code which assigns WSAE* values to $!, perl now intercepts the assignment and performs the same mapping to E* values as it uses internally when assigning to $! itself.
However, one backwards-incompatibility remains: existing Perl code which compares $! against the numeric values of the WSAE* error codes that were previously assigned to $! will now be broken in those cases where a corresponding E* value has been assigned instead. This is only an issue for those E* values < 100, which were always exported from Errno and POSIX with their original errno.h values, and therefore could not be used for WSAE* error code tests (e.g. WSAEINVAL is 10022, but the corresponding EINVAL is 22). (E* values > 100, if present, were redefined to WSAE* values anyway, so compatibility can be achieved by using the E* constants, which will work both before and after this change, albeit using different numeric values under the hood.)
The literal control form is being deprecated for two main reasons. It has what are likely unfixable bugs, such as $\cI not working as an alias for $^I, and their usage not being portable to non-ASCII platforms: While $^T will work everywhere, \cT is whitespace in EBCDIC. [perl #119123]
Setting $/ to a reference to a non integer is now forbidden and will throw an error. Perl has never documented what would happen in this context and while it used to behave the same as setting $/ to the address of the references in future it may behave differently, so we have forbidden this usage.
A warning is raised on the first call to any of them from each place in the code that they are called. (Hence a repeated statement in a loop will raise just the one warning.)
The use of interpreter-based threads in perl is officially discouraged.
The core versions of these modules will now issue "deprecated"-category warnings to alert you to this fact. To silence these deprecation warnings, install the modules in question from CPAN.
Note that the planned removal of these modules from core does not reflect a judgement about the quality of the code and should not be taken as a suggestion that their use be halted. Their disinclusion from core primarily hinges on their necessity to bootstrapping a fully functional, CPAN-capable Perl installation, not on concerns over their design.
This feature was already available in 5.18.0, but wasn't enabled by default. It is the default now, and so you no longer need build perl with the Configure argument:
-Accflags=-DPERL_NEW_COPY_ON_WRITE
It can be disabled (for now) in a perl build with:
-Accflags=-DPERL_NO_COW
On some operating systems Perl can be compiled in such a way that any attempt to modify string buffers shared by multiple SVs will crash. This way XS authors can test that their modules handle copy-on-write scalars correctly. See "Copy on Write" in perlguts for detail.
sub baz { return $cat; }
will now behave like:
sub baz { $cat; }
which is notably faster.
[perl #120765]
my $x; # or @x, %x my $y;
is now optimized to:
my ($x, $y);
In combination with the padrange optimization introduced in v5.18.0, this means longer uninitialized my variable statements are also optimized, so:
my $x; my @y; my %z;
becomes:
my ($x, @y, %z);
[perl #121077]
$OLD_PERL_VERSION was added as an alias of $].
This document was removed (actually, renamed perlgit and given a major overhaul) in Perl v5.14, causing Perl documentation websites to show the now out of date version in Perl v5.12 as the latest version. It has now been restored in stub form, directing readers to current information.
perldebguts
perlexperiment
perlfunc
perlguts
perlhack
perlhacktips
perllexwarn
perllocale
perlop
perlopentut
perlre
perlreguts
perlsub
perltrap
perlunicode
perlvar
(Actually, "OLD_PERL_VERSION" does exist, starting with this revision, but remained undocumented until perl 5.22.0.)
perlxs
(F) You used index/value array slice syntax (%array[...]) as the argument to "delete". You probably meant @array[...] with an @ symbol instead.
(F) You used key/value hash slice syntax (%hash{...}) as the argument to "delete". You probably meant @hash{...} with an @ symbol instead.
(F) You assigned a magical array to a stash element, and then tried to use the subroutine from the same slot. You are asking Perl to do something it cannot do, details subject to change between Perl versions.
New Warnings
The "auto-deref" feature is experimental.
Starting in v5.14.0, it was possible to use push, pop, keys, and other built-in functions not only on aggregate types, but on references to them. The feature was not deployed to its original intended specification, and now may become redundant to postfix dereferencing. It has always been categorized as an experimental feature, and in v5.20.0 is carries a warning as such.
Warnings will now be issued at compile time when these operations are detected.
no if $] >= 5.01908, warnings => "experimental::autoderef";
Consider, though, replacing the use of these features, as they may change behavior again before becoming stable.
Trailing white-space in a charnames alias definition is deprecated
These two deprecation warnings involving "\N{...}" were incorrectly implemented. They did not warn by default (now they do) and could not be made fatal via "use warnings FATAL => 'deprecated'" (now they can).
(W misc) A sub was declared as "sub foo : prototype(A) : prototype(B) {}", for example. Since each sub can only have one prototype, the earlier declaration(s) are discarded while the last one is applied.
(W syscalls) Embedded \0 characters in pathnames or other system call arguments produce a warning as of 5.20. The parts after the \0 were formerly ignored by system calls.
This replaces the message "Code point 0x%X is not Unicode, all \p{} matches fail; all \P{} matches succeed".
(W illegalproto) A grouping was started with "[" but never closed with "]".
(W syntax) There is a possible problem with the mixing of a control flow operator (e.g. "return") and a low-precedence operator like "or". Consider:
sub { return $a or $b; }
This is parsed as:
sub { (return $a) or $b; }
Which is effectively just:
sub { return $a; }
Either use parentheses or the high-precedence variant of the operator.
Note this may be also triggered for constructs like:
sub { 1 if die; }
(S experimental::postderef) This warning is emitted if you use the experimental postfix dereference syntax. Simply suppress the warning if you want to use the feature, but know that in doing so you are taking the risk of using an experimental feature which may change or be removed in a future Perl version:
no warnings "experimental::postderef"; use feature "postderef", "postderef_qq"; $ref->$*; $aref->@*; $aref->@[@indices]; ... etc ...
(W prototype) A prototype was declared in both the parentheses after the sub name and via the prototype attribute. The prototype in parentheses is useless, since it will be replaced by the prototype from the attribute before it's ever used.
(W syntax) In scalar context, you've used an array index/value slice (indicated by %) to select a single element of an array. Generally it's better to ask for a scalar value (indicated by $). The difference is that $foo[&bar] always behaves like a scalar, both in the value it returns and when evaluating its argument, while %foo[&bar] provides a list context to its subscript, which can do weird things if you're expecting only one subscript. When called in list context, it also returns the index (what &bar returns) in addition to the value.
(W syntax) In scalar context, you've used a hash key/value slice (indicated by %) to select a single element of a hash. Generally it's better to ask for a scalar value (indicated by $). The difference is that $foo{&bar} always behaves like a scalar, both in the value it returns and when evaluating its argument, while @foo{&bar} and provides a list context to its subscript, which can do weird things if you're expecting only one subscript. When called in list context, it also returns the key in addition to the value.
(S) exit() was called or the script otherwise finished gracefully when "PERL_EXIT_WARN" was set in "PL_exit_flags".
(S) An uncaught die() was called when "PERL_EXIT_WARN" was set in "PL_exit_flags".
(D deprecated) Using literal control characters in the source to refer to the ^FOO variables, like $^X and ${^GLOBAL_PHASE} is now deprecated. This only affects code like $\cT, where \cT is a control (like a "SOH") in the source code: ${"\cT"} and $^T remain valid.
This fixes [Perl #42957].
Unknown switch condition (?(%s in regex;
But what %s could be was mostly up to luck. For "(?(foobar))", you might have seen "fo" or "f". For Unicode characters, you would generally get a corrupted string. The message has been changed to read:
Unknown switch condition (?(...)) in regex;
Additionally, the '<-- HERE' marker in the error will now point to the correct spot in the regex.
This message is now only in the regexp category, and not in the deprecated category. It is still a default (i.e., severe) warning [perl #89648].
This warning now occurs for any %array[$index] or %hash{key} known to be in scalar context at compile time. Previously it was worded "Scalar value %%s[%s] better written as $%s[%s]".
The description for this diagnostic has been extended to cover all cases where the warning may occur. Issues with the positioning of the arrow indicator have also been resolved.
bisect.pl
The git bisection tool Porting/bisect.pl has had many enhancements.
It is provided as part of the source distribution but not installed because it is not self-contained as it relies on being run from within a git checkout. Note also that it makes no attempt to fix tests, correct runtime bugs or make something useful to install - its purpose is to make minimal changes to get any historical revision of interest to build and run as close as possible to "as-was", and thereby make "git bisect" easy to use.
find2perl
perlbug
The behaviour for all valid documented invocations is unchanged.
When compiling perl with this option, the library files for XS modules are named something "unique" -- for example, Hash/Util/Util.so becomes Hash/Util/PL_Hash__Util.so. This behavior is similar to what currently happens on VMS, and serves as groundwork for the Android port.
When building with this option set, both Configure and the compilers search for all headers and libraries under this new sysroot, instead of /.
This is a huge time saver if cross-compiling, but can also help on native builds if your toolchain's files have non-standard locations.
We now build binaries for miniperl and generate_uudmap to be used on the host, rather than running every miniperl call on the target; this means that, short of 'make test', we no longer need access to the target system once Configure is done. You can provide already-built binaries through the "hostperl" and "hostgenerate" options to Configure.
Additionally, if targeting an EBCDIC platform from an ASCII host, or viceversa, you'll need to run Configure with "-Uhostgenerate", to indicate that generate_uudmap should be run on the target.
Finally, there's also a way of having Configure end early, right after building the host binaries, by cross-compiling without specifying a "targethost".
The incompatible changes include no longer using xconfig.h, xlib, or Cross.pm, so canned config files and Makefiles will have to be updated.
For example, Android has its sh in /system/bin/sh, so if cross-compiling from a more normal Unixy system with sh in /bin/sh, "targetsh" would end up as /system/bin/sh, and "sh" as /bin/sh.
Perl 5.004 added support to use the native API of "sfio", AT&T's Safe/Fast I/O library. This code still built with v5.8.0, albeit with many regression tests failing, but was inadvertently broken before the v5.8.1 release, meaning that it has not worked on any version of Perl released since then. In over a decade we have received no bug reports about this, hence it is clear that no-one is using this functionality on any version of Perl that is still supported to any degree.
Tests now handle the errors that occur when "cygserver" isn't running.
The ALL_STATIC option has also been improved to include the Encode and Win32 extensions (for both VC++ and MinGW builds).
Support for building with EVC (Embedded Visual C++) 4 has been restored. Perl can also be built using Smart Devices for Visual C++ 2005 or 2008.
Starting in this release, almost never does application code need to distinguish between the platform's character set and Latin1, on which the lowest 256 characters of Unicode are based. New code should not use "utf8n_to_uvuni" (use "utf8_to_uvchr_buf" instead), nor "uvuni_to_utf8" (use "uvchr_to_utf8" instead),
check.third check.utf16 check.utf8 coretest minitest.prep minitest.utf16 perl.config.dashg perl.config.dashpg perl.config.gcov perl.gcov perl.gprof perl.gprof.config perl.pixie perl.pixie.atom perl.pixie.config perl.pixie.irix perl.third perl.third.config perl.valgrind.config purecovperl pureperl quantperl test.deparse test.taintwarn test.third test.torture test.utf16 test.utf8 test_notty.deparse test_notty.third test_notty.valgrind test_prep.third test_prep.valgrind torturetest ucheck ucheck.third ucheck.utf16 ucheck.valgrind utest utest.third utest.utf16 utest.valgrind
It's still possible to run the relevant commands by "hand" - no underlying functionality has been removed.
For example:
$ perl -MDevel::Peek -e'$a="abc"; $b = $a; Dump $a; Dump $b' SV = PV(0x260cd80) at 0x2620ad8 REFCNT = 1 FLAGS = (POK,IsCOW,pPOK) PV = 0x2619bc0 "abc"\0 CUR = 3 LEN = 16 COW_REFCNT = 1 SV = PV(0x260ce30) at 0x2620b20 REFCNT = 1 FLAGS = (POK,IsCOW,pPOK) PV = 0x2619bc0 "abc"\0 CUR = 3 LEN = 16 COW_REFCNT = 1
Note that both scalars share the same PV buffer and have a COW_REFCNT greater than zero.
This means that XS code which wishes to modify the "SvPVX()" buffer of an SV should call "SvPV_force()" or similar first, to ensure a valid (and unshared) buffer, and to call "SvSETMAGIC()" afterwards. This in fact has always been the case (for example hash keys were already copy-on-write); this change just spreads the COW behaviour to a wider variety of SVs.
One important difference is that before 5.18.0, shared hash-key scalars used to have the "SvREADONLY" flag set; this is no longer the case.
This new behaviour can still be disabled by running Configure with -Accflags=-DPERL_NO_COW. This option will probably be removed in Perl 5.22.
The previous behaviour can still be enabled by running Configure with -Accflags=-DPERL_SAWAMPERSAND.
The only way a NULL "sv" can be passed to sv_2*v* functions is if XS code directly calls sv_2*v*. This is unlikely as XS code uses Sv*V* macros to get the underlying value out of the SV. One possible situation which leads to a NULL "sv" being passed to sv_2*v* functions, is if XS code defines its own getter type Sv*V* macros, which check for NULL before dereferencing and checking the SV's flags through public API Sv*OK* macros or directly using private API "SvFLAGS", and if "sv" is NULL, then calling the sv_2*v functions with a NULL litteral or passing the "sv" containing a NULL value.
The public API newATTRSUB was previously a macro to the private function Perl_newATTRSUB. Function Perl_newATTRSUB has been removed. newATTRSUB is now macro to a different internal function.
This bottom level function decodes the first character of a UTF-8 string into a code point. It is accessible to "XS" level code, but it's discouraged from using it directly. There are higher level functions that call this that should be used instead, such as "utf8_to_uvchr_buf" in perlapi. For completeness though, this documents some changes to it. Now, tests for malformations are done before any tests for other potential issues. One of those issues involves code points so large that they have never appeared in any official standard (the current standard has scaled back the highest acceptable code point from earlier versions). It is possible (though not done in CPAN) to warn and/or forbid these code points, while accepting smaller code points that are still above the legal Unicode maximum. The warning message for this now includes the code point if representable on the machine. Previously it always displayed raw bytes, which is what it still does for non-representable code points.
Many flags that used to be exposed via regexp.h and used to populate the extflags member of struct regexp have been removed. These fields were technically private to Perl's own regexp engine and should not have been exposed there in the first place.
The affected flags are:
RXf_NOSCAN RXf_CANY_SEEN RXf_GPOS_SEEN RXf_GPOS_FLOAT RXf_ANCH_BOL RXf_ANCH_MBOL RXf_ANCH_SBOL RXf_ANCH_GPOS
As well as the follow flag masks:
RXf_ANCH_SINGLE RXf_ANCH
All have been renamed to PREGf_ equivalents and moved to regcomp.h.
The behavior previously achieved by setting one or more of the RXf_ANCH_ flags (via the RXf_ANCH mask) have now been replaced by a *single* flag bit in extflags:
RXf_IS_ANCHORED
pluggable regex engines which previously used to set these flags should now set this flag ALONE.
utf8::upgrade( my $u = "\x{e5}"); utf8::downgrade(my $d = "\x{e5}"); /$u$d/
[RT #118297]
This could occur for pseudo-forked processes too, as Win32's pseudo-fork is implemented in terms of threads. [perl #77672]
Greg McCarroll died on August 28, 2013.
Greg was well known for many good reasons. He was one of the organisers of the first YAPC::Europe, which concluded with an unscheduled auction where he frantically tried to raise extra money to avoid the conference making a loss. It was Greg who mistakenly arrived for a london.pm meeting a week late; some years later he was the one who sold the choice of official meeting date at a YAPC::Europe auction, and eventually as glorious leader of london.pm he got to inherit the irreverent confusion that he had created.
Always helpful, friendly and cheerfully optimistic, you will be missed, but never forgotten.
Excluding auto-generated files, documentation and release tools, there were approximately 280,000 lines of changes to 1,800 .pm, .t, .c and .h files.
Perl continues to flourish into its third decade thanks to a vibrant community of users and developers. The following people are known to have contributed the improvements that became Perl 5.20.0:
Aaron Crane, Abhijit Menon-Sen, Abigail, Abir Viqar, Alan Haggai Alavi, Alan Hourihane, Alexander Voronov, Alexandr Ciornii, Andy Dougherty, Anno Siegel, Aristotle Pagaltzis, Arthur Axel 'fREW' Schmidt, Brad Gilbert, Brendan Byrd, Brian Childs, Brian Fraser, Brian Gottreu, Chris 'BinGOs' Williams, Christian Millour, Colin Kuskie, Craig A. Berry, Dabrien 'Dabe' Murphy, Dagfinn Ilmari Mannsaaker, Daniel Dragan, Darin McBride, David Golden, David Leadbeater, David Mitchell, David Nicol, David Steinbrunner, Dennis Kaarsemaker, Dominic Hargreaves, Ed Avis, Eric Brine, Evan Zacks, Father Chrysostomos, Florian Ragwitz, Francois Perrad, Gavin Shelley, Gideon Israel Dsouza, Gisle Aas, Graham Knop, H.Merijn Brand, Hauke D, Heiko Eissfeldt, Hiroo Hayashi, Hojung Youn, James E Keenan, Jarkko Hietaniemi, Jerry D. Hedden, Jess Robinson, Jesse Luehrs, Johan Vromans, John Gardiner Myers, John Goodyear, John P. Linderman, John Peacock, kafka, Kang-min Liu, Karen Etheridge, Karl Williamson, Keedi Kim, Kent Fredric, kevin dawson, Kevin Falcone, Kevin Ryde, Leon Timmermans, Lukas Mai, Marc Simpson, Marcel Gruenauer, Marco Peereboom, Marcus Holland-Moritz, Mark Jason Dominus, Martin McGrath, Matthew Horsfall, Max Maischein, Mike Doherty, Moritz Lenz, Nathan Glenn, Nathan Trapuzzano, Neil Bowers, Neil Williams, Nicholas Clark, Niels Thykier, Niko Tyni, Olivier Mengue, Owain G. Ainsworth, Paul Green, Paul Johnson, Peter John Acklam, Peter Martini, Peter Rabbitson, Petr PisaX, Philip Boulain, Philip Guenther, Piotr Roszatycki, Rafael Garcia-Suarez, Reini Urban, Reuben Thomas, Ricardo Signes, Ruslan Zakirov, Sergey Alekseev, Shirakata Kentaro, Shlomi Fish, Slaven Rezic, Smylers, Steffen Mueller, Steve Hay, Sullivan Beck, Thomas Sibley, Tobias Leich, Toby Inkster, Tokuhiro Matsuno, Tom Christiansen, Tom Hukins, Tony Cook, Victor Efimov, Viktor Turskyi, Vladimir Timofeev, YAMASHINA Hio, Yves Orton, Zefram, Zsban Ambrus, AEvar Arnfjoerd` Bjarmason.
The list above is almost certainly incomplete as it is automatically generated from version control history. In particular, it does not include the names of the (very much appreciated) contributors who reported issues to the Perl bug tracker.
Many of the changes included in this version originated in the CPAN modules included in Perl's core. We're grateful to the entire CPAN community for helping Perl to flourish.
For a more complete list of all of Perl's historical contributors, please see the AUTHORS file in the Perl source distribution.
If you believe you have an unreported bug, please run the perlbug program included with your release. Be sure to trim your bug down to a tiny but sufficient test case. Your bug report, along with the output of "perl -V", will be sent off to<EMAIL_ADDRESS>to be analysed by the Perl porting team.
If the bug you are reporting has security implications, which make it inappropriate to send to a publicly archived mailing list, then please send it to<EMAIL_ADDRESS>This points to a closed subscription unarchived mailing list, which includes all the core committers, who will be able to help assess the impact of issues, figure out a resolution, and help co-ordinate the release of patches to mitigate or fix the problem across all platforms on which Perl is supported. Please only use this address for security issues in the Perl core, not for modules independently distributed on CPAN.
The INSTALL file for how to build Perl.
The README file for general stuff.
The Artistic and Copying files for copyright information.
| 2019-10-21 | perl v5.30.3 |