Zend Framework
LICENSE
This source file is subject to the new BSD license that is bundled with this package in the file LICENSE.txt. It is also available through the world-wide-web at this URL: http://framework.zend.com/license/new-bsd If you did not receive a copy of the license and are unable to obtain it through the world-wide-web, please send an email to<EMAIL_ADDRESS>so we can send you a copy immediately.
enableRequestDebugLogging(
string $logfile
)
:
This method enables logging of requests by changing the Zend_Http_Client_Adapter used for performing the requests.
NOTE: This will not work if you have customized the adapter already to use a proxy server or other interface.
generateIfMatchHeaderData(
$data, $allowWeek
)
:
string|null
Returns the data for an If-Match header based on the current Etag property. If Etags are not supported by the server or cannot be extracted from the data, then null will be returned.
getNextFeed(
Zend_Gdata_App_Feed $feed, string $className
=
null
)
:
Zend_Gdata_App_Feed|null
Retrieve next set of results based on a given feed.
getPreviousFeed(
Zend_Gdata_App_Feed $feed, string $className
=
null
)
:
Zend_Gdata_App_Feed|null
Retrieve previous set of results based on a given feed.
retrieveAllEntriesForFeed(
object $feed
)
:
mixed
Retrieve all entries for a feed, iterating through pages as necessary.
Be aware that calling this function on a large dataset will take a significant amount of time to complete. In some cases this may cause execution to timeout without proper precautions in place.
useObjectMapping(
boolean $value
)
:
void
Enable/disable the use of XML to object mapping.
usingObjectMapping(
)
:
boolean
Determine whether service object is using XML to object mapping.
Provides Atom Publishing Protocol (APP) functionality. This class and all other components of Zend_Gdata_App are designed to work independently from other Zend_Gdata components in order to interact with generic APP services.
DEFAULT_MAJOR_PROTOCOL_VERSION = '1'
Default major protocol version.
DEFAULT_MINOR_PROTOCOL_VERSION = 'null'
Default minor protocol version.
string $_defaultPostUri = 'null'
Default URI to which to POST.
boolean $_gzipEnabled = 'false'
Enable gzipped responses?
Zend_Http_Client $_httpClient = ''
Client object used to communicate
boolean $_httpMethodOverride = 'false'
Override HTTP PUT and DELETE request methods?
$_majorProtocolVersion = ''
Indicates the major protocol version that should be used.
At present, recognized values are either 1 or 2. However, any integer value >= 1 is considered valid.
Under most circumtances, this will be automatically set by Zend_Gdata_App subclasses.
int $_maxRedirects = '5'
Maximum number of redirects to follow during HTTP operations
$_minorProtocolVersion = ''
Indicates the minor protocol version that should be used. Can be set to either an integer >= 0, or NULL if no minor version should be sent to the server.
At present, this field is not used by any Google services, but may be used in the future.
Under most circumtances, this will be automatically set by Zend_Gdata_App subclasses.
array $_registeredPackages = 'array'
Packages to search for classes when using magic __call method, in order.
Zend_Http_Client $_staticHttpClient = 'null'
Client object used to communicate in static context
boolean $_useObjectMapping = 'true'
Whether we want to use XML to object mapping when fetching data.
boolean $_verboseExceptionMessages = 'true'
Use verbose exception messages. In the case of HTTP errors, use the body of the HTTP response in the exception message.
__call(
string $method, array $args
)
:
Provides a magic factory method to instantiate new objects with shorter syntax than would otherwise be required by the Zend Framework naming conventions. For instance, to construct a new Zend_Gdata_Calendar_Extension_Color, a developer simply needs to do $gCal->newColor(). For this magic constructor, packages are searched in the same order as which they appear in the $_registeredPackages array
__construct(
Zend_Http_Client $client
=
null, string $applicationId
=
MyCompany-MyApp-1.0
)
:
Create Gdata object
delete(
mixed $data, $remainingRedirects
=
null
)
:
void
DELETE entry with client object
get(
string $uri, array $extraHeaders
=
array
)
:
Zend_Http_Response
GET a URI using client object.
getEntry(
string $uri, string $className
=
Zend_Gdata_App_Entry
)
:
string|Zend_Gdata_App_Entry
Retrieve entry as string or object
getFeed(
string $uri, string $className
=
Zend_Gdata_App_Feed
)
:
string|Zend_Gdata_App_Feed
Retrieve feed as string or object
getGzipEnabled(
)
:
boolean
Get the HTTP override state
getHttpClient(
)
:
Zend_Http_Client
Get the Zend_Http_Client object used for communication
getHttpMethodOverride(
)
:
boolean
Get the HTTP override state
getMajorProtocolVersion(
)
:
int
Get the major protocol version that is in use.
getMaxRedirects(
)
:
int
Get the maximum number of redirects to follow during HTTP operations
getMinorProtocolVersion(
)
:
(int|NULL)
Get the minor protocol version that is in use.
getStaticHttpClient(
)
:
Zend_Http_Client
Gets the HTTP client object. If none is set, a new Zend_Http_Client will be used.
getVerboseExceptionMessages(
)
:
boolean
Get whether to use verbose exception messages
In the case of HTTP errors, use the body of the HTTP response in the exception message.
import(
string $uri, Zend_Http_Client $client
=
null, string $className
=
Zend_Gdata_App_Feed
)
:
string|Zend_Gdata_App_Feed
Imports a feed located at $uri.
importFile(
string $filename, string $className
=
Zend_Gdata_App_Feed, string $useIncludePath
=
false
)
:
Zend_Gdata_App_Feed
Imports a feed from a file located at $filename.
importString(
string $string, string $className
=
Zend_Gdata_App_Feed, $majorProtocolVersion
=
null, $minorProtocolVersion
=
null
)
:
Zend_Gdata_App_Feed
Imports a feed represented by $string.
importUrl(
string $url, string $className
=
Zend_Gdata_App_Feed, array $extraHeaders
=
array
)
:
string|Zend_Gdata_App_Feed
Imports the specified URL (non-statically).
insertEntry(
mixed $data, string $uri, string $className
=
Zend_Gdata_App_Entry, array $extraHeaders
=
array
)
:
Zend_Gdata_App_Entry
Inserts an entry to a given URI and returns the response as a fully formed Entry.
performHttpRequest(
string $method, string $url, array $headers
=
null, string $body
=
null, string $contentType
=
null, int $remainingRedirects
=
null
)
:
Zend_Http_Response
Performs a HTTP request using the specified method
post(
mixed $data, string $uri
=
null, $remainingRedirects
=
null, string $contentType
=
null, array $extraHeaders
=
null
)
:
Zend_Http_Response
POST data with client object
prepareRequest(
string $method, string $url
=
null, array $headers
=
array, mixed $data
=
null, string $contentTypeOverride
=
null
)
:
array
Provides pre-processing for HTTP requests to APP services.
put(
mixed $data, string $uri
=
null, $remainingRedirects
=
null, string $contentType
=
null, array $extraHeaders
=
null
)
:
Zend_Http_Response
PUT data with client object
registerPackage(
string $name
)
:
void
Adds a Zend Framework package to the $_registeredPackages array.
This array is searched when using the magic __call method below to instantiante new objects.
setGzipEnabled(
boolean $enabled
=
false
)
:
void
Toggle requesting gzip encoded responses
setHttpClient(
Zend_Http_Client $client, $applicationId
=
MyCompany-MyApp-1.0
)
:
Zend_Gdata_App
Set the Zend_Http_Client object used for communication
setHttpMethodOverride(
boolean $override
=
true
)
:
void
Toggle using POST instead of PUT and DELETE HTTP methods
Some feed implementations do not accept PUT and DELETE HTTP methods, or they can't be used because of proxies or other measures. This allows turning on using POST where PUT and DELETE would normally be used; in addition, an X-Method-Override header will be sent with a value of PUT or DELETE as appropriate.
setMajorProtocolVersion(
int $value
)
:
Set the major protocol version that should be used. Values < 1 will cause a Zend_Gdata_App_InvalidArgumentException to be thrown.
setMaxRedirects(
int $maxRedirects
)
:
void
Set the maximum number of redirects to follow during HTTP operations
setMinorProtocolVersion(
(int|NULL) $value
)
:
Set the minor protocol version that should be used. If set to NULL, no minor protocol version will be sent to the server. Values < 0 will cause a Zend_Gdata_App_InvalidArgumentException to be thrown.
setStaticHttpClient(
Zend_Http_Client $httpClient
)
:
void
Set the static HTTP client instance
Sets the static HTTP client object to use for retrieving the feed.
setVerboseExceptionMessages(
boolean $verbose
)
:
Set whether to use verbose exception messages
In the case of HTTP errors, use the body of the HTTP response in the exception message.
updateEntry(
mixed $data, string|null $uri
=
null, string|null $className
=
null, array $extraHeaders
=
array
)
:
Zend_Gdata_App_Entry
Update an entry


Choose a category below. If you answer the question correctly, you will be awarded 50 points. For each incorrect answer, you lose 10 points. If you get the question wrong, you can try again but you only have 10 turns to answer before the game is over so it may be best to move on to a new question! Remember: "The way forward is sometimes the way back."


ECO-AFRICA Conference (EFC) is one of leading African forums on the green economy. It brings together world -class experts around Africa to directly focus on advancing the global green economy and sustainability agenda, achieving the UN Sustainable Development Goals and implementing the recommendations of COP 21 & 22.
ECO-AFRICA is the premier international platform to drive dialogue and action among industry stakeholders, and build partnerships and collaboration by connecting creative minds across different sectors. The summit will showcases innovative solutions and leading practices to inspire change, and aims to align stakeholders with policy makers to create a facilitative environment and ecosystem for sustainable green growth that address current and future needs.
It is a critical need to address now, Air pollution in Nigeria kills more than 6 million people every year, is the biggest environmental health risk of our time. Airborne pollutants are responsible for about one third of deaths from stroke, chronic respiratory disease, and lung cancer, as well as one quarter of deaths from heart attack. Air pollution is also fundamentally altering our climate, with profound impacts on the health of the planet.
Transport activity is increasing around Nigerian cities as economies grow, which means that the sector’s emissions are also on the rise due to high demand of over population. That’s largely because 95 per cent of the transportation system energy still comes from fossil fuels.
Nigerian seas and oceans are facing unprecedented threats as a result of human activity more especially in Niger Delta Region. Every year, an estimated of million tonnes of chemicals and plastic waste end up in Nigerian oceans. At the same time, climate change is damaging coral reefs and other key ecosystems; leaving the water untreated.


circadian []. virus viral clock immune expression infection cells {border-style HCV px;} rhythms #ffffff;} protein mice molecular replication biological different BMAL cellular gene core human infected response rhythmic time transcription disease genes hepatitis peripheral processes PER cell levels DNA changes control infections influenza innate promoter px; {font-family Acad. Circadian Natl. RNA Sci. [,]. clocks components font-size histone increased influence interplay melatonin reported Clock USA./pnas. [,,]. adaptive antiviral core-clock double; effect epigenetic genotype hepatocellular host impact inflammatory mouse oscillators role transcriptional treatment vaccination PER, Rev. Viral airway body border-color border-width caused central clockwork decreased factors function genetic interaction life liver lung main mammalian nucleus oscillations outcome performed pulmonary regulation regulator responses showed shown solid; study suprachiasmatic temperature timing variation virus, viruses {font-style {font-weight Biological Bmal CLOCK, CRY, Huh- ICP, REV-ERBα SCN Virus [], activation activity antibody bold chromatin clock-controlled controls courier; driven efficacy elements em; environmental expressing functioning ganglion immunity interesting killer linked lipid locomotor lymph mechanisms methylation model modifications mortality natural observed pathological physiological pt;} recent retinal retinohypothalamic rhythmicity studies survival various


Synchronous call
Asynchronous call
Sync call without argument
Async call without argument
echo.syn
echo.asyn
Stress test，2K times consecutive asynchronous API calls
Never call because without @JavascriptInterface annotation
( This test is
just for Android ,should be ignored in IOS )
call progress
hasNativeMethod("xx")
hasNativeMethod("testSyn")


Objective:
The goal is to create a salary estimator based on years of experience of the employees of an anonymous company, using a simple Linear Regression Model build from scratch applied on “Salary_Data.csv” Kaggle’s dataset. Then I will compare the performance of the model against the prebuilt Sklearn linear regression model.
I won’t explain theory here, only background basic equations needed for code understanding.
Table of contents
- Dataset
- Linear model definition
- Loss function definition
- Gradient definition
- Gradient descent algorithm
- Training the model on data
- Model evaluation
- Salary estimation
- Comparison with Sklearn model
- Conclusion
Importing libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
1.Dataset
The dataset is constituted of “YearsExperience” and “Salary” informations for 30 employees in an anonymous company.
Get the original dataset here: https://www.kaggle.com/karthickveerakumar/salary-data-simple-linear-regression
Importing dataset
df = pd.read_csv("data/Salary_Data.csv", delimiter= ',')
A simple dataframe visualization using markdown format:
import tabulate
print(df.head(7).to_markdown())
| | YearsExperience | Salary |
|---:|------------------:|---------:|
| 0 | 1.1 | 39343 |
| 1 | 1.3 | 46205 |
| 2 | 1.5 | 37731 |
| 3 | 2 | 43525 |
| 4 | 2.2 | 39891 |
| 5 | 2.9 | 56642 |
| 6 | 3 | 60150 |
A better visualization for this blog is done by exporting dataframe visualization to png format:
#!pip install dataframe-image
import dataframe_image as dfi
# I export the dataframe visualization as png
# so I can use markdown to show down the dataframe
dfi.export(df.head(7), "img/linear_regression_dataframe.png")
Data visualization
#setting plots background color
plt.rcParams.update({'axes.facecolor':'#f8fafc'})
df.plot.scatter('YearsExperience', 'Salary',
figsize=(10,6), color='darkorange',
marker='o', edgecolor='k', s=100)
plt.xlabel('Years Experience', fontsize = 12)
plt.ylabel('Annual Salary (USD)', fontsize = 12)
plt.title('Annual Salary v/s Years of Experience', fontsize = 15)
plt.show()
One can see on plot that annual salaries of employees looks linearly correlated to years of experience !
2.Linear model definition
In statistics, linear regression is a linear approach to modelling the relationship between a scalar response and one or more explanatory variables (also known as dependent and independent variables). The case of one explanatory variable is called simple linear regression.
Let’s define y as the real random variable to be explained (dependent) and x the explanatory variable (independent). The model supposes that on average, y is an affine function of x. The writing of the model implicitly supposes a preliminary notion of causality in the sense that y depends on x:
For a given sequence of m observations
$${(x_i, y_i) \hspace{4mm}i=1 …, m}$$
$$ y_i=\beta_{0} +\beta_{1}.x_i + \epsilon_i $$
$$ Using \hspace{3mm}matrix \hspace{3mm}notations \hspace{3mm}x = [x_1, x_2, …, x_m]^T \hspace{3mm} and \hspace{3mm} y = [y_1, y_2, …, y_m]^T$$
$$The \hspace{3mm}model \hspace{3mm}is:\hspace{3mm} y = X.\beta + \epsilon$$
$$Where\hspace{3mm} X = [1 \hspace{3mm} x] \hspace{3mm}and\hspace{3mm} \beta= [\beta_{0} \hspace{3mm}\beta_{1}]^T$$
The goal is to find the parameters which would provide the “best” fit in some sense for the data points:
$$ \hat{y_i}=\hat{\beta_{0}} +\hat{\beta_{1}}.x_i $$
Converting data into arrays
To do calculations, we need to convert our data into arrays that will feed our custom model…
x = df["YearsExperience"].values.reshape(-1, 1)
y = df["Salary"].values.reshape(-1, 1)
print(x.shape, y.shape)
(30, 1) (30, 1)
…And add identity column vector to feature:
$$X = [1 \hspace{3mm} x]$$
in order to do the product:
$$X.\beta \hspace{3mm}where\hspace{3mm}\beta= [\beta_{0} \hspace{3mm}\beta_{1}]^T$$
X = np.hstack((np.ones(x.shape), x))
X[0:5]
array([[1. , 1.1],
[1. , 1.3],
[1. , 1.5],
[1. , 2. ],
[1. , 2.2]])
Defining model
def model(X, Beta):
return np.matmul(X, Beta)
Parameters initialization
For visualization purpose, I added a constant value to Beta initial values.
Beta = np.random.randn(2, 1) + 10000
print(Beta)
[[10000.63920093]
[ 9999.95848202]]
Checking model’s initial state
y_init = model(X, Beta)
plt.figure(figsize=(10,6))
plt.plot(x, y_init, label='Custom model initial state',
c= 'purple', marker='None', linestyle= 'solid')
plt.scatter(x, y, c= 'darkorange', label='Dataset', marker='o', edgecolor='k', s=100)
plt.legend(facecolor='white')
plt.title('Annual Salary v/s Years of Experience', fontsize = 15)
plt.xlabel('Years Experience', fontsize = 12)
plt.ylabel('Annual Salary (USD)', fontsize = 12)
plt.xticks(np.arange(1, 12, 1))
plt.yticks(np.arange(20000, 140000, 10000))
plt.grid(linestyle='--')
plt.show()
One clearly see that the model is not yet fitted to our data.
3.Loss function definition using Mean Squared Error (MSE)
From the previous equations, we know that the residuals are defined by:
$$ \epsilon = y - X.\beta$$
Then to measure the error of our model we define the Loss function as the mean of the squares of the residuals:
$$ J(\beta) = \frac{1}{m} \sum (y - X.\beta)^2 $$
def lossFunction(X, y, Beta):
m = y.shape[0] # where m is the number of samples
return 1/(m) * np.sum(np.square(y - model(X, Beta)))
The Loss is the value that we want to minimize in order to find the best regression line. Let’s check the initial Loss value:
print(lossFunction(X, y, Beta))
199243474.14642814
Notice the high value of our Loss !! It can be possible to scale the values of the x feature in order to reduce the high values during the calculations … but for this try we can stay with non scaled values.
4.Gradient definition
Now we calculate the gradient of the Loss function:
$$\frac{\partial J(\beta) }{\partial \beta} = - \frac{2}{m} X^T.(y - X.\beta)$$
But as we want to solve a minimization problem, we keep for calculations the gradient multiplied by -1:
def gradient(X, y, Beta):
m = y.shape[0]
return 2/m * np.matmul(X.T, model(X, Beta) - y)
Minimizing the Loss function means that its final gradient should be close to zero. Let’s check the initial values of the gradient:
print(gradient(X, y, Beta))
[[ -25738.49612919]
[-128193.59810319]]
5.Gradient descent algorithm
For each iteration of the calculus we update the parameters as well as the gradient of the model using the gradient descent algorithm:
$$\beta' = \beta - \alpha \frac{\partial J(\beta) }{\partial \beta}$$
def gradientDescent(X, y, Beta, learning_rate, n_iterations):
loss_history = np.zeros(n_iterations)
for i in range(n_iterations):
Beta = Beta - learning_rate * gradient(X, y, Beta)
loss_history[i] = lossFunction(X, y, Beta)
return Beta, loss_history, gradient(X, y , Beta)
6.Training the model on data
Fisrt I define the hypermarameters:
N_ITERATIONS = 20000
LEARNING_RATE = 0.001
Then I train the model on the data…
Final_Beta, loss_history, Final_gradient = gradientDescent(X, y, Beta, LEARNING_RATE, N_ITERATIONS)
… check the final values of the gradient (closer to zero as expected !) …
print(Final_gradient)
[[-1.38763209]
[ 0.20591879]]
… and get the final regression coefficients:
print(f"Intercept = {Final_Beta[0][0]:.2f}")
print(f"coefficient = {Final_Beta[1][0]:.2f}")
Intercept = 25788.92
coefficient = 9450.45
Let’s have a look on the Learning Curve:
plt.figure(figsize=(10,6))
plt.plot(range(N_ITERATIONS), loss_history, c= 'purple')
plt.title('Learning curve', fontsize = 15)
plt.xlabel('Iterations', fontsize = 12)
plt.ylabel('Loss value', fontsize = 12)
plt.show()
7.Model evaluation
Regression line visualization
y_pred_custom = model(X, Final_Beta)
Let’s have a look on the regression line:
plt.figure(figsize=(10,6))
plt.plot(x, y_pred_custom, label='Custom model prediction',
c= 'purple', marker='None', linestyle= 'solid')
plt.scatter(x, y, c= 'darkorange', label='Dataset', marker='o', edgecolor='k', s=100)
plt.legend(facecolor='white', loc="upper left")
plt.title('Annual Salary v/s Years of Experience', fontsize = 15)
plt.xlabel('Years Experience', fontsize = 12)
plt.ylabel('Annual Salary (USD)', fontsize = 12)
plt.xticks(np.arange(1, 12, 1))
plt.yticks(np.arange(30000, 140000, 10000))
plt.grid(linestyle='--')
plt.show()
We can see on plot that this time the model fits pretty good on the data !
Coefficient of determination
This coefficient indicates the quality of predictions made with our model.
$$ R^2 = 1 - \frac{\sum ((y_i - \hat{y_i})^2}{\sum ((y_i - \bar{y})^2)} $$
def determinationCoef(y, y_pred):
SSR = ((y - y_pred)**2).sum() # Residual sum of squares SSR
SST = ((y - y.mean())**2).sum() # Total sum of squares SST
return np.sqrt(1 - SSR/SST)
print(f"R2 = {determinationCoef(y, y_pred_custom):0.2f}")
R2 = 0.98
We almost reach the maximum value of 1 !! The model can estimate accuratly the annual salary of an employee in the company given his years of experience.
8.Salary estimation
Now let’s imagine that we want to have an estimation of the salary for someone who have 12 years of experience in the company.
x_estimation = np.array([12])
X_estimation = np.hstack((np.ones(x_estimation.shape), x_estimation))
y_estimation = model(X_estimation, Final_Beta)
print(f'The estimated annual salary of someone who has 12 years of experience is \
{y_estimation[0]:.0f} USD \n... according to our dataset.')
The estimated annual salary of someone who has 12 years of experience is 139194 USD
... according to our dataset.
Let’s visualize the result on the plot:
plt.figure(figsize=(10,6))
plt.scatter(x_estimation, y_estimation, label='Estimation',
c= 'yellowgreen', marker='o', edgecolor='k', s=200)
plt.plot(x, y_pred_custom, label='Custom model prediction',
c= 'purple', marker='None', linestyle= 'solid')
plt.scatter(x, y, c= 'darkorange', label='Dataset', marker='o', edgecolor='k', s=100)
plt.legend(facecolor='white', loc="upper left")
plt.title('Annual Salary v/s Years of Experience', fontsize = 15)
plt.xlabel('Years Experience', fontsize = 12)
plt.ylabel('Annual Salary (USD)', fontsize = 12)
plt.xticks(np.arange(1, 13, 1))
plt.yticks(np.arange(30000, 150000, 10000))
plt.grid(linestyle='--')
plt.show()
One can see on plot the estimated salary for someone who has 12 years of experience in the company.
9.Comparison with sklearn LinearRegression model
Scikit-Learn already provides a well built Linear Regression model for general purpose (linear, multi-linear or polynomial regressions).
I will apply it on the Salary_data.csv dataset in order to compare and check the quality of my custom linear regression model.
As I’ve done previously, I define X, y variables
X = df.iloc[:, :-1].values
y = df.iloc[:, -1].values.reshape(-1, 1)
print(X.shape, y.shape)
(30, 1) (30, 1)
Next I define a linear regression model using Sklearn library…
from sklearn.linear_model import LinearRegression
LR_model = LinearRegression()
…fit the model to the data…
LR_model.fit(X, y)
LinearRegression()
…and get the resulting regression coefficients
print(f"Intercept = {LR_model.intercept_[0]:.2f}")
print(f"coefficient = {LR_model.coef_[0][0]:.2f}")
Intercept = 25792.20
coefficient = 9449.96
Now let’s see how well the model fits to the data
y_pred_sklearn = LR_model.predict(X)
plt.figure(figsize=(10,6))
plt.plot(X, y_pred_sklearn, label='Sklearn model prediction',
c= 'darkturquoise', marker='None', linestyle= 'solid')
plt.scatter(X, y, c= 'darkorange', label='Dataset', marker='o', edgecolor='k', s=100)
plt.legend(facecolor='white', loc="upper left")
plt.title('Annual Salary v/s Years of Experience', fontsize = 15)
plt.xlabel('Years Experience', fontsize = 12)
plt.ylabel('Annual Salary (USD)', fontsize = 12)
plt.xticks(np.arange(1, 12, 1))
plt.yticks(np.arange(30000, 140000, 10000))
plt.grid(linestyle='--')
plt.show()
Salary estimation:
X_estimation = np.array([12]).reshape(-1,1)
y_estimation = LR_model.predict(X_estimation)
print(f'The estimated annual salary of someone who has 12 years of experience is \
{y_estimation[0][0]:.0f} USD')
The estimated annual salary of someone who has 12 years of experience is 139192 USD
10.Conclusion
The custom model build from scratch looks pretty good to predict salary given years of experience and fits pretty well to the data, thus I get a R2 determination coefficient of 0.98 (almost 1) that indicates a good quality of predictions.
Comparing coefficients of regression and estimated salaries for both models, one can see that the custom model gives the same results as the Sklearn model.
import tabulate
import pandas as pd
results = {
'Model': ["Custom LR model", 'Sklearn LR model'],
'Intercetp': [25788.92, 25792.20],
'coefficient': [9450.45, 9449.96],
'12 Years exp. estimated Ann. Salary': [139194, 139192]}
df_results = pd.DataFrame(data=results).set_index('Model')
print(df_results.to_markdown())
| Model | Intercetp | coefficient | 12 Years exp. estimated Ann. Salary |
|:-----------------|------------:|--------------:|--------------------------------------:|
| Custom LR model | 25788.9 | 9450.45 | 139194 |
| Sklearn LR model | 25792.2 | 9449.96 | 139192 |
Now we have a rather precise tool which can help human resources to estimate the annual salary of an employee of this anonymous company according to his number of experience in this company.


Rate Your Professor
Log in
Sign-up
Log in
Log in
account_circle
Email
lock
Password
Sign in
Forgot password ?
Click here


This page shows model predictions for a particular customer from the data set. The full list of features created from aggregations of the customer transaction history are also summarized in the "Model Features" table.
| {{ col }} | {% endfor %}
|---|
| {{val}} | {% endfor %}
| {{ col }} | {% endfor %}
|---|
| {{val}} | {% endfor %}
| {{ col }} | {% endfor %}
|---|
| {{val}} | {% endfor %}


UK tribunal dismisses appeal against ruling that all NI citizens are automatically born British
The tribunal made its initial ruling following an appeal by the Home Office in October.
by Stephen McDermottTHE UK’S UPPER Tribunal has refused an application by Derry woman Emma DeSouza to appeal a ruling that people born in the North are automatically British citizens.
It follows a ruling by the tribunal in October, when the Home Office appealed a previous ruling by a lower immigration tribunal that de Souza was an “Irish national only who has only ever been such”.
The case came about after DeSouza applied for a residence card for her US-born husband Jake in 2015, but was told she had to apply as a British citizen because she was unable to do so as an Irish citizen.
DeSouza argued that she had never considered herself British, so could not renounce citizenship she never had, and also insisted that the Home Office position is contrary to the 1998 Good Friday Agreement.
However, UK government lawyers successfully argued that the British Nationality Act – not the Good Friday Agreement – was the relevant legislation. DeSouza announced at the time that she would appeal that ruling.
But today, the Upper Tribunal refused her application for an appeal, with Judge Rintoul stating that “the Belfast Agreement manifestly is not” a constitutional document.
Speaking about the latest ruling, DeSouza said the tribunal had failed to engage on the significance of Good Friday Agreement and the impact of its decision in October.
“We are deeply disappointed to be suffering a further set back. This decision will cost us both substantial hardship and expense as we now prepare to appeal directly to the Court of Appeal,” she added.
The Department of Foreign Affairs and Trade has previously said that Tánaiste Simon Coveney would raise the issue with the Secretary of State for Northern Ireland.
The Home Office also previously said that it is committed to upholding the Good Friday Agreement and that it respects the right of the people of Northern Ireland to choose to identify as British or Irish or both.
With reporting from Gráinne Ní Aodha.


{% block head %}{% endblock %}
2Do by zszazi
2Do App
Simple App to note down the ToDo Tasks
by zszazi
{% block body %}{% endblock %}


THINGS I'VE LEARNED
INTERNET BASICS
HTTP requests
IP address
Servers
HTML
Stands for
Hyper Text Markup Language
A ton of tags for formatting pages
Boilerplate tags
!DOCTYPE html
html
head
title
body
Headings: h1 - h6
Paragraph: p
em
strong


Dean Smith in Alfredo Morelos Rangers confession as Aston Villa boss namechecks in-form striker
The 23-year-old has scored 21 goals this season and Aston Villa are understood to be interested.
by Mark McDougall, https://www.dailyrecord.co.uk/authors/mark-mcdougall/Aston Villa boss Dean Smith has admitted he kept a close eye on Alfredo Morelos as the striker downed Porto.
The Colombian opened the scoring for Rangers with a stunning strike before setting up Steven Davis for the second.
Aston Villa have been heavily linked with a move for Morelos and Smith concedes he had a watching brief of the striker's exploits.
But he insists he won't speculate further on the 21-goal frontman.
He said: "I certainly saw his goal last night.
"It was a very good finish.
"He's a player who is contracted to another football club and we aren't a club that talks about them."
Morelos has also been linked with a move to Crystal Palace while the likes of Frankfurt are also believed to be keen.
Steven Gerrard insists he isn't going anywhere in the January window despite a rush of recent rumours.
That's despite Rangers posting an £11m financial loss. Morelos has often spoken about his wish to move to the English Premier League.
Gerrard had said: "I'm not confident, I'm sure, I'm 100 per cent sure he's not going anywhere."


Visit our site at http://www.entertainmentcruises.com
HERE’S ANOTHER REASON TO CRUISE MYSTIC BLUE THIS YEAR…
Get 20% off ANY Mystic Blue cruise now through December 31, 2014.*
Save on: - Brunch Cruises - Lunch Cruises - Dinner Cruises - Holiday
Cruises like July 4th, New Year’s Eve and more
Use promo code CYBER to save.Hurry, you must book before 1/15/14!
Book Now
(http://reservations.entertainmentcruises.com/search/?metro=Chicago&pax=1-19&tod=Any&product=Mystic_d%/14)
|
Pricing
(http://reservations.entertainmentcruises.com/search/?metro=Chicago&pax=1-19&tod=Any&product=Mystic_d%/14)
|
Directions (http://www.mysticbluecruises.com/Chicago/directions) |
FAQs
(http://www.mysticbluecruises.com/Chicago/frequently-asked-questions)
About Us
(http://www.entertainmentcruises.com/Entertainment-Cruises/aboutus )
| Privacy Policy
(http://www.entertainmentcruises.com/Entertainment-Cruises/privacypolicy
) | Our Service System
(http://www.entertainmentcruises.com/Entertainment-Cruises/our-service-system
) | Contact Us
(http://www.entertainmentcruises.com/Entertainment-Cruises/contactus
) | Email Sign-up (http://mysticbluecruises.com/Chicago/email-sign-up
)
Mystic Blue Cruises Chicago | 401 East Illinois Street, Suite 310 |
Chicago, IL 60611
Use promo code CYBER to save. View HTML online here
(http://archives.subscribermail.com/msg/09c73eae57a241fd857deb6f30de375e.htm).
Get 20% off ANY Cruise in 2014!
*Must book before 1/15/14. Valid for parties of 1-19. Limited seats
available on select events. Valid on cruises through 12/31/14. Cannot
be combined with other offers. Subject to availability.
Safely Unsubscribe
http://app.subscribermail.com/unsub.cfm?tempid=09c73eae57a241fd857deb6f30de375e&mailId=52e165471d4e49dfa7ffeb6f30de375e
Send this mail to your friend.
http://app.subscribermail.com/send_friend.cfm?template=09c73eae57a241fd857deb6f30de375e&mailid=52e165471d4e49dfa7ffeb6f30de375e
Powered by SubscriberMail http://www.subscribermail.com
Entertainment Cruises401 E. Illinois StreetChicago, Illinois 60611


Development workflow¶
You already have your own forked copy of the networkx repository, by following Making your own copy (fork) of networkx. You have Set up your fork. You have configured git by following Configure git. Now you are ready for some real work.
Workflow summary¶
In what follows we’ll refer to the upstream networkx master
branch, as
“trunk”.
- Don’t use your
master
branch for anything. Consider deleting it. - When you are starting a new set of changes, fetch any changes from trunk, and start a new feature branch from that.
- Make a new branch for each separable set of changes — “one task, one branch” (ipython git workflow).
- Name your branch for the purpose of the changes - e.g.
bugfix-for-issue-14
orrefactor-database-code
. - If you can possibly avoid it, avoid merging trunk or any other branches into your feature branch while you are working.
- If you do find yourself merging from trunk, consider Rebasing on trunk
- Ask on the networkx mailing list if you get stuck.
- Ask for code review!
This way of working helps to keep work well organized, with readable history. This in turn makes it easier for project maintainers (that might be you) to see what you’ve done, and why you did it.
See linux git workflow and ipython git workflow for some explanation.
Consider deleting your master branch¶
It may sound strange, but deleting your own master
branch can help reduce
confusion about which branch you are on. See deleting master on github for
details.
Update the mirror of trunk¶
First make sure you have done Linking your repository to the upstream repo.
From time to time you should fetch the upstream (trunk) changes from github:
git fetch upstream
This will pull down any commits you don’t have, and set the remote branches to
point to the right commit. For example, ‘trunk’ is the branch referred to by
(remote/branchname) upstream/master
- and if there have been commits since
you last checked, upstream/master
will change after you do the fetch.
Make a new feature branch¶
When you are ready to make some changes to the code, you should start a new branch. Branches that are for a collection of related edits are often called ‘feature branches’.
Making an new branch for each set of related changes will make it easier for someone reviewing your branch to see what you are doing.
Choose an informative name for the branch to remind yourself and the rest of us
what the changes in the branch are for. For example add-ability-to-fly
, or
buxfix-for-issue-42
.
# Update the mirror of trunk
git fetch upstream
# Make new feature branch starting at current trunk
git branch my-new-feature upstream/master
git checkout my-new-feature
Generally, you will want to keep your feature branches on your public github
fork of networkx. To do this, you git push this new branch up to your
github repo. Generally (if you followed the instructions in these pages, and by
default), git will have a link to your github repo, called origin
. You push
up to your own repo on github with:
git push origin my-new-feature
In git >= 1.7 you can ensure that the link is correctly set by using the
--set-upstream
option:
git push --set-upstream origin my-new-feature
From now on git will know that my-new-feature
is related to the
my-new-feature
branch in the github repo.
The editing workflow¶
Overview¶
# hack hack
git add my_new_file
git commit -am 'NF - some message'
git push
In more detail¶
Make some changes
See which files have changed with
git status
(see git status). You’ll see a listing like this one:# On branch ny-new-feature # Changed but not updated: # (use "git add <file>..." to update what will be committed) # (use "git checkout -- <file>..." to discard changes in working directory) # # modified: README # # Untracked files: # (use "git add <file>..." to include in what will be committed) # # INSTALL no changes added to commit (use "git add" and/or "git commit -a")
Check what the actual changes are with
git diff
(git diff).Add any new files to version control
git add new_file_name
(see git add).To commit all modified files into the local copy of your repo,, do
git commit -am 'A commit message'
. Note the-am
options tocommit
. Them
flag just signals that you’re going to type a message on the command line. Thea
flag — you can just take on faith — or see why the -a flag? — and the helpful use-case description in the tangled working copy problem. The git commit manual page might also be useful.To push the changes up to your forked repo on github, do a
git push
(see git push).
Ask for your changes to be reviewed or merged¶
When you are ready to ask for someone to review your code and consider a merge:
Go to the URL of your forked repo, say
https://github.com/your-user-name/networkx
.Use the ‘Switch Branches’ dropdown menu near the top left of the page to select the branch with your changes:
Click on the ‘Pull request’ button:
Enter a title for the set of changes, and some explanation of what you’ve done. Say if there is anything you’d like particular attention for - like a complicated change or some code you are not happy with.
 This is still a good way of getting some preliminary code review.
Some other things you might want to do¶
Delete a branch on github¶
git checkout master
# delete branch locally
git branch -D my-unwanted-branch
# delete branch on github
git push origin :my-unwanted-branch
Note the colon :
before my-unwanted-branch
. See also:
https://help.github.com/articles/pushing-to-a-remote/#deleting-a-remote-branch-or-tag
Several people sharing a single repository¶
If you want to work on some stuff with other people, where you are all committing into the same repository, or even the same branch, then just share it via github.
First fork networkx into your account, as from Making your own copy (fork) of networkx.
Then, go to your forked repository github page, say
https://github.com/your-user-name/networkx
Click on the ‘Admin’ button, and add anyone else to the repo as a collaborator:
Now all those people can do:
git clone git@githhub.com:your-user-name/networkx.git
Remember that links starting with git@
use the ssh protocol and are
read-write; links starting with git://
are read-only.
Your collaborators can then commit directly into that repo with the usual:
git commit -am 'ENH - much better code'
git push origin master # pushes directly into your repo
Explore your repository¶
To see a graphical representation of the repository branches and commits:
gitk --all
To see a linear list of commits for this branch:
git log
You can also look at the network graph visualizer for your github repo.
Finally the Fancy log output lg
alias will give you a reasonable text-based
graph of the repository.
Rebasing on trunk¶
Let’s say you thought of some work you’d like to do. You
Update the mirror of trunk and Make a new feature branch called
cool-feature
. At this stage trunk is at some commit, let’s call it E. Now
you make some new commits on your cool-feature
branch, let’s call them A, B,
C. Maybe your changes take a while, or you come back to them after a while. In
the meantime, trunk has progressed from commit E to commit (say) G:
A---B---C cool-feature
/
D---E---F---G trunk
At this stage you consider merging trunk into your feature branch, and you remember that this here page sternly advises you not to do that, because the history will get messy. Most of the time you can just ask for a review, and not worry that trunk has got a little ahead. But sometimes, the changes in trunk might affect your changes, and you need to harmonize them. In this situation you may prefer to do a rebase.
rebase takes your changes (A, B, C) and replays them as if they had been made to
the current state of trunk
. In other words, in this case, it takes the
changes represented by A, B, C and replays them on top of G. After the rebase,
your history will look like this:
A'--B'--C' cool-feature
/
D---E---F---G trunk
See rebase without tears for more detail.
To do a rebase on trunk:
# Update the mirror of trunk
git fetch upstream
# go to the feature branch
git checkout cool-feature
# make a backup in case you mess up
git branch tmp cool-feature
# rebase cool-feature onto trunk
git rebase --onto upstream/master upstream/master cool-feature
In this situation, where you are already on branch cool-feature
, the last
command can be written more succinctly as:
git rebase upstream/master
When all looks good you can delete your backup branch:
git branch -D tmp
If it doesn’t look good you may need to have a look at Recovering from mess-ups.
If you have made changes to files that have also changed in trunk, this may generate merge conflicts that you need to resolve - see the git rebase man page for some instructions at the end of the “Description” section. There is some related help on merging in the git user manual - see resolving a merge.
Recovering from mess-ups¶
Sometimes, you mess up merges or rebases. Luckily, in git it is relatively straightforward to recover from such mistakes.
If you mess up during a rebase:
git rebase --abort
If you notice you messed up after the rebase:
# reset branch back to the saved point
git reset --hard tmp
If you forgot to make a backup branch:
# look at the reflog of the branch
git reflog show cool-feature
8630830 cool-feature@{0}: commit: BUG: io: close file handles immediately
278dd2a cool-feature@{1}: rebase finished: refs/heads/my-feature-branch onto 11ee694744f2552d
26aa21a cool-feature@{2}: commit: BUG: lib: make seek_gzip_factory not leak gzip obj
...
# reset the branch to where it was before the botched rebase
git reset --hard cool-feature@{2}
Rewriting commit history¶
Note
Do this only for your own feature branches.
 Or perhaps the you made several false starts you would like the posterity not to see.
This can be done via interactive rebasing.
Suppose that the commit history looks like this:
git log --oneline
eadc391 Fix some remaining bugs
a815645 Modify it so that it works
2dec1ac Fix a few bugs + disable
13d7934 First implementation
6ad92e5 * masked is now an instance of a new object, MaskedConstant
29001ed Add pre-nep for a copule of structured_array_extensions.
...
and 6ad92e5
is the last commit in the cool-feature
branch. Suppose we
want to make the following changes:
- Rewrite the commit message for
13d7934
to something more sensible. - Combine the commits
2dec1ac
,a815645
,eadc391
into a single one.
We do as follows:
# make a backup of the current state
git branch tmp HEAD
# interactive rebase
git rebase -i 6ad92e5
This will open an editor with the following text in it:
pick 13d7934 First implementation
pick 2dec1ac Fix a few bugs + disable
pick a815645 Modify it so that it works
pick eadc391 Fix some remaining bugs
# Rebase 6ad92e5..eadc391 onto 6ad92e5
#
# Commands:
# p, pick = use commit
# r, reword = use commit, but edit the commit message
# e, edit = use commit, but stop for amending
# s, squash = use commit, but meld into previous commit
# f, fixup = like "squash", but discard this commit's log message
#
# If you remove a line here THAT COMMIT WILL BE LOST.
# However, if you remove everything, the rebase will be aborted.
#
To achieve what we want, we will make the following changes to it:
r 13d7934 First implementation
pick 2dec1ac Fix a few bugs + disable
f a815645 Modify it so that it works
f eadc391 Fix some remaining bugs
This means that (i) we want to edit the commit message for
13d7934
, and (ii) collapse the last three commits into one. Now we
save and quit the editor.
Git will then immediately bring up an editor for editing the commit message. After revising it, we get the output:
[detached HEAD 721fc64] FOO: First implementation
2 files changed, 199 insertions(+), 66 deletions(-)
[detached HEAD 0f22701] Fix a few bugs + disable
1 files changed, 79 insertions(+), 61 deletions(-)
Successfully rebased and updated refs/heads/my-feature-branch.
and the history looks now like this:
0f22701 Fix a few bugs + disable
721fc64 ENH: Sophisticated feature
6ad92e5 * masked is now an instance of a new object, MaskedConstant
If it went wrong, recovery is again possible as explained above.


Queries CRM for team membership of specified user and copies them to USD data parameters as one string


{% include head.html %} {% include nav.html %}
Hello my name is Adam and I help businesses realize opportunities in the digital space. I work primarily in the Healthcare and Education industries but welcome opportunities in all fields. My previous work includes developing educational products and services for Healthcare clients and the public schools.
I learned to program as a kid and have used that skill to realize business opportunties, develop products for customers and to automate internal processes. Education and Training is an area where I developed great communication and presentation skills necessary to administer training programs as well.
I hold both Bachelor's and Master's degree in Education and Information systems respectively. In my free time I'm often practicing with programming new algorithims or running a local meetup that allows me the opportunity to develop as a speaker, and leader.


This class define a dynamic graph wrapper around the vicon client.
More...
#include <ros_robot_state_publisher.hpp>
|
|
| RosRobotStatePublisher (const std::string &name) |
| |
| void | add (const std::string &base_link_name, const std::string &joint_names, const std::string &tf_prefix, const std::string &signal_name, const std::string &joint_state_topic_name) |
| |
| int & | trigger (int &, int) |
| | Trigger the publishing of the data to ros for all signals. More...
|
| |
|
void | send_data (const RosRobotStatePublisherInternal &publisher, int time) |
| | Signal callback functions.
|
| |
|
|
static const std::string | CLASS_NAME |
| | The class name used to identify it in the dynamic graph pool.
|
| |
This class define a dynamic graph wrapper around the vicon client.
◆ add()
| void dynamic_graph::RosRobotStatePublisher::add |
( |
const std::string & |
base_link_name, |
|
|
const std::string & |
joint_names, |
|
|
const std::string & |
tf_prefix, |
|
|
const std::string & |
signal_name, |
|
|
const std::string & |
joint_state_topic_name |
|
) |
| |
- Parameters
-
| base_link_name | |
| joint_names | |
| tf_prefix | |
| signal_name | |
◆ getClassName()
| virtual const std::string& dynamic_graph::RosRobotStatePublisher::getClassName |
( |
void |
| ) |
const |
|
inlineprivatevirtual |
Get the CLASS_NAME object.
- Returns
- const std::string&
◆ normalize_tf_msg_quaternion()
| void dynamic_graph::RosRobotStatePublisher::normalize_tf_msg_quaternion |
( |
geometry_msgs::TransformStamped & |
msg | ) |
|
|
private |
Normalize the tf message quaternion before sending it to the.
- Parameters
-
◆ set_tf_msg_to_identity()
| void dynamic_graph::RosRobotStatePublisher::set_tf_msg_to_identity |
( |
geometry_msgs::TransformStamped & |
msg | ) |
|
|
private |
Set a tf msg to identity.
- Parameters
-
◆ trigger()
| int & dynamic_graph::RosRobotStatePublisher::trigger |
( |
int & |
dummy, |
|
|
int |
time |
|
) |
| |
Trigger the publishing of the data to ros for all signals.
- Returns
- int& dummy stuff.
◆ publishers_
This is the list of publishers registered for this class.
It correspond to the list of robot states one wants to publish. See the add methode (Command)
◆ trigger_signal_
| SignalOUT dynamic_graph::RosRobotStatePublisher::trigger_signal_ |
|
private |
Create an internal output signal which should be call periodically in the device in order to publish the data.
The callback function of this signal is the trigger methode.
The documentation for this class was generated from the following files:


OBWriteDomesticStandingOrder3_Data_Initiation_CreditorAccount
- Identification : Identification
- Name : String
The account name is the name or names of the account owner(s) represented at an account level. Note, the account name is not the product name or the nickname of the account. OB: ASPSPs may carry out name validation for Confirmation of Payee, but it is not mandatory.
- SchemeName : OBExternalAccountIdentification4Code
- SecondaryIdentification : SecondaryIdentification


Hi,
I would like to post on emoncms some data sent by my electric meter over the "teleinfo" protocol. In France, most of "recent" meters have this feature. More info here about that protocol.
I found a schematic and a sketch (attached) to get those data on a arduino uno. So I just need to modify this sketch to post the data over RFM12B. It should not be problematic. This sketch use a software serial port to talk with the meter. So it needs a digital input. D8 is used in the schematic with a pull up resistor.
I just need to know which pin may I use on an EmonTH. Same question for an emonTX V3.
For both, if possible, I would like to avoid using a pin also needed for something else (pulse/temperature)
Eric
EDIT : I saw the port map in the technical wiki but anyway ... I have a lack of knowledge to answer my question
I would like to be be sure to not damage my TH/TX
Re: Teleinfo on EmonTH/TX
hi
i tried to make a minimal sketch just to get you going... its one of the example sketches but shaved to actyally only flash the led
you need to get the data out and put the data in variables....
typedef struct {
int realPower; // RealPower
tells it what to transmit.... so right now the sketch will just send 0 or some bogus number since the variable is not set... it will transmit every 10 sec..
hope it helps a bit
Re: Teleinfo on EmonTH/TX
Hi Eric,
Probably stating the obvious when I say the problem of not using a pre-assigned pin is physical access, only the pins assigned to temp or pulse are brought to a connector.
The emonTx v3 however does have a pin accessible next to the existing terminals DIO9 http://wiki.openenergymonitor.org/index.php?title=File:EmonTxV3-io.png or there is DIO10 on the RHS below the ISP conn.
I don't think the emonTH is as easy but DIO10 is not used (DIO9 is LED on TH) and is the 2nd pin from the bottom of the row of 10 connections on the LHS https://raw.githubusercontent.com/openenergymonitor/Hardware/master/emonTH/photo.jpg so you could attach a small fly lead.
Paul
Re: Teleinfo on EmonTH/TX
ups... total misunderstood the Q....
http://solderpad.com/openenergymon/emonth/
shows the schematic...
if not much mistaken dig 5, 6 ,9 are taken, but dig 2 & 3 seems free...
Re: Teleinfo on EmonTH/TX
Thank's both.
Well, I can't read the schematics of the EmonTH/TX. (Lack of knowledge in electronics)
Let's first discuss about the pins assigned to temp or pulse as there are easily accessible on the terminal block.
Can I use them knowing that I will add a 4.7k pull-up resistor as needed in my schematic ?
For the others pins,
- Emontx : only the dig 10 seems easily accessible (dig 9 is not connected)
- Emonth : no pin is easily accessible ?
Eric
Re: Teleinfo on EmonTH/TX
i only looked at schematics for emonth.... there are more free pins on emontx... or at least they are all accessible
http://solderpad.com/openenergymon/emontx/
look at P1-4.... follow the green lines that goes to the AVR. Pins 11-13 are free...
http://darcy.rsgc.on.ca/ACES/TEI3M/images/BreadboardArduinoIntroPics063.png shows both the names of the pins as atmel defines them in datasheet but also their names in the arduino world...
Re: Teleinfo on EmonTH/TX
hmm but yes you can use the temp and pulse also if you dont use them for anything else...
so if you say have a emonth near the meter but the DHT sensor is not fitted you can use that...
or you can use the pulse input...
so if your meterbox is not getting direct sunlight and have a white surface i would use the pulse input and fit the DHT sensor so i can log both temp and electricity...
and if you are lucky that you own the place i would make sure there is some ventilation to the inside of the meter box
Re: Teleinfo on EmonTH/TX
It's unclear about DIO 9 the comment is crossed out in the port map but the track is connected in the schematic and pcb layout. The schematics for the RFu-328 on ciscos site also show DIO9 connected.
If you want a clean pin (no other componants to remove) then DIO10 (or 9) is favorite I guess.
DIO2 (pulse) has a 10K pull-down res, the temp terminals cannot be used on emonTH if an on-board temp sensor fitted.
If you were to use the temp conn on the emonTx v3 there is a 4K7 resistor across the terminals so by setting digitalpin 19 high (just as you would if using a temp sensor) the circuit would be as per your diagram minus the LED and only be 3.3v not 5v.
Care should be taken regards power consumption as there not much headroom for the led but the opto-coupler should be ok, may need to recalc the res value if there's no led and using 3.3v not 5v I don't know how critical it is,
I'm sure Robert will be able to advise you better on this.
Paul
Re: Teleinfo on EmonTH/TX
Thank you Paul for raising the power consumption issue.
This is very important as I would like to use a EmonTH rather than an EmonTx.
Eric
Re: Teleinfo on EmonTH/TX
I am guessing that there is enough current in the opto-coupler emitter to saturate the detector, so we assume that it will drop around 0.25 V (0.4 V max). The blue LED will have a high forward voltage, so I suggest replacing it with a standard red one, with a forward voltage of about 1.7 V at low current, normally about 2 V. With the supply at the nominal 3.3 V, that leaves about 1 V for the resistor. If the LED current is 5 mA (you can afford 9 mA maximum before the emonTx V3 power supply will malfunction at minimum mains voltage), then the series resistor needs to be 200 Ω. The original circuit runs the LED at about 0.3 mA, if that gives acceptable brightness, the resistor value for that is 3.3 kΩ.
If you are running an emonTH on battery, it would be best to omit the LED and use as high a value resistor as possible - you should be able to use up to 470 kΩ with no problem.
Re: Teleinfo on EmonTH/TX
Thank's Robert.
I will give a try with that conditions:
- using an emonTH on battery
- omitting the led
- without fitting any on-board temp sensor nor using external temp sensors
In that conditions :
- can I use the DIg2 (pulse) knowing it has a 10K pull-down resistor ?
- can I use the Dig19 (1-Wire Data) ?
Eric
Eric
Re: Teleinfo on EmonTH/TX
I think D2 is the best solution.
Although D2 has R2 (10 kΩ) shown as a pull-down to GND, that resistor is not present unless you are using the DS18B20. (It is a wire-ended through-hole component under the battery case, at the corner nearest to the aerial.)
Therefore, you can use the terminal block for all the connections. Connect your pull-up resistor between 4 (D2) & 2 (3.3 V) and the wires to your opto-coupler between 4 & 3 (GND)
Re: Teleinfo on EmonTH/TX
Ok great !
I will try as soon as I receive the needed components.
If it works, I will share my sketch.
Bye
Re: Teleinfo on EmonTH/TX
Eric,
I don't know how is your setup, but just in case, If you have any linux device (Pi, Wifi-Fi Router, ...) near your Teleinfo output you can try the teleinfo broadcast that send all teleinfo data to your local network and/or to emoncms also.
Re: Teleinfo on EmonTH/TX
Hi Charly,
Thank's for this information but I would rather use an EmonTH rather than an Linux machine to capture the teleinfo input. (The meter is out of my home).
I made some tests and it works great as I manage to print the teleinfo packets to the serial monitor.
Now, I need to write a sketch to send the data to my EmonBase.
Eric


a grammar for binary file formats
| with a growing number of binary formats that are being used, there is a
need for specifying these formats in a well-defined way. context free grammars have been
used to specify the syntax of programming langauges. to use a grammar for binary file
formats seems to be a logical choice. in this page such a grammar, named bff, is described. it has several construct that are not traditionally found in context free grammars for programming language. due to the nature of binary file formats, it is important to be able to reference information that has been read before. for example, a string of characters might be preceded by a number that indicates the lengt of the string. the terminal symbols of the grammar consist of a number of bytes, representing one of the basic data types, such as: char, short int, long int, float and double. differences in byte ordering for integers, and the different formats for floating point numbers should be taken into account. due to the nature of binary formats, it is not too restrictive to use only recursive descent grammars, e.i., grammars that can be parsed top-down, and belong to the ll(1) class. the bff is tested for the dwg file format. as we start with this file format, bff will naturally first focus on the requirements based on this format, and because of this it will be slightly biased. to specify the grammar of bff we will use a form of extended bnf. tool support for bff the first tool i am thinking about is a program that can read the grammar and apply this to a given binary file, resulting in an annotated output. with this tool should also support the reverse engineering of binary file formats. in a later state, a tool could be made that generates a parser and the needed data structures for reading a binary file into memory according to the grammar. as it is not always required (nor possible) to read the whole file into memory, it should be possible to generate procedures to read the file interactively. the form of the bff grammar a grammar that describes a binary file format consists of the specification of the elementary units of data, and the rules by which these should be grouped together. the elementary units we assume that a binary file can be viewed as a stream of bytes (as this is the most commonly used unit of data). usually a number of bytes are grouped together to form data values that cannot be represented by a single byte. to specify a word value consisting of two bytes, for example, we propose the following defintion style: type word := byte : first, byte : second return ((word)first | ((word)second << 8)). a word representation where the lower order byte comes before the higher order is usually used by small endian processors. the expression used on will be based on c. we assume that the following types have been defined on top of the default types of c: typedef unsigned char byte; typedef unsigned short int word; typedef unsigned long int longword; this leeds us to the definition of the basic types that will be supported: c_data_types := "char" | "byte" | "short" | "word" | "long" | "longword" | "float" | "double" . (we assume for the moment that float and double represent floating numbers of 4, respectively 10 bytes.) the grammar of the rule used for defining types is: type_def_rule := "typedef" c_data_type basic_type_name ":=" ("byte" ":" byte_name) list "return" expr "." . here expr stands for c-like expression using the byte_names as they are used in the rule. the basic_type_names should not be confused with the c_type_names. it is possible that the same name is both used as a basic_type_name and a c_type_name. the rules the grammar that specifies in which order elementary units are taken from a binary file, makes use of non-terminal symbols and rules for each non-terminal symbol. there will be one non-terminal symbol that will parse the whole binary file, which will be called the root non-terminal symbol. for each non-terminal symbol there has to be a rule describing the elements it consists of, where each element is either an elementary elements or a non-terminal symbols. the rule of the root non-terminal symbol comes as the first rule, and is preceded with the word `root'. the whole bff grammar follows the following grammar: bff_grammar := type_def_rule seq "root" rule rule seq. each rule has a non-terminal symbol on the left-hand side, and a list of elements on the right-hand side. each element is either a elementary element, a non-terminal symbol, or a grouping of elements. because bff assumes a top-down parsing method, it is possible to give each non-terminal symbol a number of parameters. this leads to the following grammar for the rules: rule := non_term_name ( "(" param list ")" )opt ":=" elem list ".". each element consist of the following parts: * an (optional) range, which specifies the range of the file that element may read. * a data type, which can either be a elementary unit, a non-terminal symbol, or a list of elements enclosed by brackets. * an (optional) times expression, to indicate that the element can be repeated, either for a given number of times or for an unknown number of times. * an (optional) identifying name, which can be used later to reference the value found. * an (optional) equivalence expression, which can be used for checking. the following grammar describes an element: elem := range opt data_type ( "[" expr "]" | "*" ) ( ":" elem_name )opt ( "=" c_expr )opt. range := "[" file_pos (":" file_pos)opt "]". file_pos := "begin" | "end" | "cur" | expr. data_type := "(" elem list ")" | basic_type_name | non_term_name ( "(" expr list ")" )opt. |


Apps
45 south ave
600 ft
0°F
interior
0°F
exterior
author
title
album
Big Buck Bunny
9:23 AM
John Johnson
9:45 AM
Cheryl Johnson
10:01 AM
CNN
0
mph
0%
0 mi


Improve the user experience: make registration easy and convenient from admins to attendees.
Build user-friendly, customizable, and ad-free online registration forms that integrate with your website.
Manage everything online: Payment history, participant info, required documents, reports, and more.
Colleen Bell
Wieuca Road Baptist Church Summer Camps


Employees
Improved productivity and satisfaction as your employees stay informed, connected and can feed requests and updates in real time
Effectively and quickly from your mobile device. Staying connected has never been easier with zonifero.
Improved productivity and satisfaction as your employees stay informed, connected and can feed requests and updates in real time
IoT sensors provide real-time information about occupancy and help employees in a quick way to find currently available/free desks and conference rooms.
Powerful booking system implemented in Zonifero improved by IOT sensors provides the possibility to manage your conference rooms, hot-desks, and parking spots. Thanks to the Zonifero your employees will be able to make a quick reservation by one smart app.
Zonifero will help your company resolve many prolems in your office.
Zonifero will help your company resolve many prolems in your office.
Zonifero will help your company resolve many prolems in your office.
Find hot desks in a breeze. Book any workplace at your office locations around the world. Do you need a parking space? Just a few clicks and its done.
zonifero business supports meeting rooms booking. Reservations can be made from the smartphone, tablet in front of the room or directly from the calendar in Office365 or G Suite. Room reservations can also be made in companies that are not familiar with Office 365 or G Suite services.
unlimited rooms
unlimited employees
10 floors
Room booking
Hot-desk booking
Status of employees
Request management
Office map
Parking spots booking
Notifications
FAQ module
Integration with calendar
List of employees


Well, well... Look at all this fearmongering!
Christmas has come early, and we're outta here...
So, here's our post-analysis and data dump.
PS - To Discord
All you had to do is prevent CSRF on the login endpoint, but you decided to simply block our server's IP address from connecting from your site - which clearly didn't work.
Please do better next time...


This is a simple template for a html5 canvas animation
The project is made by Gianluca Guarini and you can feedback on github
The core of this application is main.js. In this file you can find every basic objects you need to start a great canvas animation.
As you know to draw on a canvas you need to create a javascript loop and update the coordinates of every object on the stage. The most important thing to know is that you can loop in three ways:
- Using the new window.requestAnimationFrame API
- Using setInterval()
- Using setTimeout() at the end of the loop function calling itself again and again
Here you can find some examples of this ways to loop and as you should know you can choose the best way to animate your application being careful to CPU performance


In this post, I'll talk about how I solved the HackerRank Triangle Numbers problem the long and why. In this problem, the triangle of numbers is created as follows:
1
1 1 1
1 2 3 2 1
1 3 6 7 6 3 1
Each number in any level is the sum of the number directly above it and its neighbors (not its own neighbors, the neighbors of the number directly above). A non-existent neighbor is counted as a 0, i.e. the first two and last two numbers in any row will only be the sum of one or two numbers. The problem asks you to give the position of the first even number a given level of the triangle. No analytical idea popped at out at me immediately, so I decided with a brute force solution. Maybe I would have some insight while breaking down the recursive formula for generating the levels
Next Level Function
First I made a function that produced a new level from a given level called next_level. I also imported the deque object from the built-in collections module. On a side note, I picked up deque from Python Cookbook, which I highly recommend. Among other things, you can set it to contain only a certain number of elelemnts, so that when you add an element to the end, it pops out the element at its head. This is really convenient when you want to traverse some array but only look at some fixed length subarray. Here I'll need to traverse the "current level" looking every subaray of 3 consequtive elements (to get the sums to calculate values for the level below).
from collections import deque def next_level(current_level):
Each new level has a 1 on either end automatically. The next numbers in are the sum of the first two and last two values of the current level respectively. I refer to them as "left end" and "right end."
l_end = current_level[0] + current_level[1] r_end = current_level[-1] + current_level[-2]
That leaves a "middle section" with 2 elements shorter than the current level. We'll make an empty middle section container to hold these values. We'll also make a deque to traverse our current level. The deque will have a max length of 3, and be initially populated by the first 3 elements of the current level. This way we can calculate the deque sum to find the value the corresponding element for the middle, then append the next current level element to the deque, calculate the deque sum for the next middle element, etc. etc. until we have all of our values. The try/except here is so that I don't have to worry about indexing, which can be annoying. I can let the deque keep going until there are no more middle elements left uncalculated without telling it explicitely where to stop.
middle = [None]*(len(a)-2) temp = deque(a[0:3], maxlen=3) for i in range(len(middle)): middle[i] = sum(temp) try: temp.append(current_level[i+3]) except: pass
Then finally putting the entire next level together, adding the 1's on either end, we get
return [1] + [l_end] + middle + [r_end] + [1]
Specific Level Function
Next, I needed a function to find any specific level. I could have modified my previous function to be recursive, but I chose a simpler method for me to implement, which was to create another function that called the next_level function. This function would be called produce_level and take an integer x representing the level of the triangle to produce. These two functions together would act like a single recursive function. The produce_level function would start at level 2 which is [1,1,1] and keeping calling next_level on subsequent levels, keeping track of what level it was at, and stopping at the level asked for
def produce_level(level_number): '''produces the given level of the number triangle''' current_level_number = 2 current_level = [1,1,1] while level < level_number: current_level = next_level(current_level) current_level_number += 1 return current_level
First Even Number Position Function
Finally I would make a third function that would go through a given level and find the place of the first even number. This could have easily been added to the previous function, but it was just simpler to think about all of these pieces modularly. This function would basically go through the array starting at the beginning and find the first element n, where n/2 had a remainder of 0. I would use the modulo function to do this (% in python; e.g. 17%3 = 2 since 17/3 = 5 remainder 2). It would return the index+1 because in python the first element is element number 0, while in the problem, the first element is considered element number 1.
def find_starting_even(level_number): '''finds the position of first even number in the given number triangle level''' current_level = produce_level(level_number) for index, element in enumerate(current_level): if element%2 == 0: return index+1 else: pass
Test Running
Now I wanted to test run my functions to see if they worked. Plus maybe the output would reveal some kind of pattern. By the way, I wan't worried about robustness of inputs, and I knew I would only be looking for levels above 2 and that Iw ould be the only one using these functions and for only a very short period of time. Here was the output for levels 3 through 7
for i in range(3, 8): produce_level(i) find_starting_even(i)
Produced
[1, 2, 3, 2, 1] 2 [1, 3, 6, 7, 6, 3, 1] 3 [1, 4, 10, 16, 19, 16, 10, 4, 1] 2 [1, 5, 15, 30, 45, 51, 45, 30, 15, 5, 1] 4 [1, 6, 21, 50, 90, 126, 141, 126, 90, 50, 21, 6, 1] 2
Everything checks out
Searching for a Pattern
Everything is looking good, but I doubt HackerRank would let me solve this by brute force. Their testing aapparatus tends to time out if you solve problems using brute force, and when I tried this, it did infact timeout. So I had to find some sort of pattern. Instead of printing out a bunch of outputs and sorting through it, I figured a pattern would be easiest to detect visually. So I used a loop would generate a list of the first 50 values output by the find_starting_even function (starting with level 3), make a list to hold the values, and then make a graph from that list.
import matplotlib.pyplot as plt starting_even_position = [] for i in range(3,54): starting_even_position.append(find_starting_even(i)) x = range(3,54) y = starting_even_position plt.plot(x,y) plt.show()
Here is the resulting graph created:
The Simplest Solution from the Longest Route
The graph shows us that the first even number is generated by a nice clear and simple pattern: 2,3,4,3,2,3,4,3,2,3... Now all we need to know is if the traingle level we are looking for is divisible by 2 and 4 to instantly know the position of the first even number:
def first_even_position(level_number): if level_number%2: return 2 elif level_number%4: return 4 else: return 3
This simple function passed with flying colors.
Conclusion
This was the long way to solve this problem for sure. I made more functions than I needed to, I used brute force as a starting point, I made graphs that revealed a pattern I probably could have picked up on through simple inspection, etc. but why? Well in doing this problem I realized that I optimized for myself and I really didn't mind. I had fun doing it this way, and I didn't go searching for some silver bullet algorithm I could apply, I didn't bother myself with being overly clever or performing intense mental gymnastics. I just jumped in and kept everything simple at the expense of a little initial efficieny. This apporach isn't always appropriate, but I definitely don't mind when it is.


Monklands' Specsavers branches don pink outfits for Breast Cancer awareness day
Breast Cancer Now is the UK’s largest breast cancer charity and dedicated to funding research.
by Ian Bunting, https://www.dailyrecord.co.uk/authors/ian-bunting/Eyes were firmly fixed on fundraising fun as a Monklands opticians took part in Breast Cancer Now’s wear it pink day.
The teams at Specsavers in Airdrie and Coatbridge all donned the colour pink and decorated the stores in blush colours.
Funds were raised by donations to charity buckets and the collections raised £52.
Breast Cancer Now is the UK’s largest breast cancer charity and dedicated to funding research.
Click here for more from Airdrie and Coatbridge
The charity believes that, by acting now, by 2050 everyone who develops breast cancer will live; but fundraising is vital to make it happen.
Andrew McGonagle, store director at Specsavers Airdrie and Coatbridge, said: ‘It was great fun to take part in wear it pink again this year with everyone getting on board to raise funds and awareness for Breast Cancer Now.
“A big thank you goes to everyone who donated; I’m delighted to be able to present these funds to this very worthy cause.”


Home Accessory Architect
Installer v2.1
JSON Config:
Remove WiFi Settings
> Auto OTA Updates
No networks found
{% for network in networks %}
{{ network.ssid }}
{% endfor %}
Enter SSID manually
SSID:
Password:
by José A. Jiménez Campos


We are Harmonize. A Blockchain based Sound Sharing System, that is ideal for artists who are just beginning. Harmonize was created by considering the state of artists who have to pay money to third party people, and did not want to do it
Bootstrap 4 has been noted as one of the most reliable and proven frameworks and Mobirise has been equipped to develop websites using this framework.
Responsive
One of Bootstrap 4's big points
One of Bootstrap 4's big points is responsiveness and Mobirise makes effective use of this by generating highly responsive website for you.
Web Fonts
Google has a highly exhaustive list of fonts
Google has a highly exhaustive list of fonts compiled into its web font platform and Mobirise makes it easy for you to use them on your website easily and freely.
Unlimited Sites
Mobirise gives you the freedom to develop
Mobirise gives you the freedom to develop as many websites as you like given the fact that it is a desktop app.
SHARE THIS PAGE!


Chapter 7 Data Wrangling: tidyr
7.1 Overview
Now you have some experience wrangling and working with tidy data. But we all know that not all data that you have are tidy. So how do we make data more tidy? With the tidyr
package.
Objectives
- learn
tidyr
with thegapminder
package - other wrangling: joins, binding
- practice the RStudio-GitHub workflow
- your turn: use the data wrangling cheat sheet to explore window functions
Resources
These materials borrow heavily from:
7.1.1 Setup
We’ll work today in RMarkdown. You can either continue from the same RMarkdown as yesterday, or begin a new one.
Here’s what to do:
- Clear your workspace (Session > Restart R)
- New File > R Markdown…, save as something other than
gapminder-wrangle.Rmd
and delete irrelevant info, or just continue usinggapminder-wrangle.Rmd
I’m going to write this in my R Markdown file:
Data wrangling with `tidyr`, which is part of the tidyverse. We are going to tidy some data!
7.2 Finishing up dplyr
7.3 Joining datasets
Sometimes you have data coming from different places or in different files, and you want to put them together so you can analyze them. This is called relational data, because it has some kind of relationship. In the tidyverse, combining data that has a relationship is called “joining”.
From the RStudio cheatsheet (note: this is an earlier version of the cheatsheet but I like the graphics):
Notice how you may not have exactly the same observations in the x1 columns above. Observations A and B are the same, but notice how the table on the left has observation C, and the table on the right has observation D.
If you wanted to combine these two tables, how would you do it? There are some decisions you’d have to make about what was important to you. The cheatsheet visualizes it for us:
We will just talk about this briefly here, but you can refer to this more as you have your own datasets that you want to join. This describes the figure above::
left_join
keeps everything from the left table and matches as much as it can from the right table. In R, the first thing that you type will be the left table (because it’s on the left)right_join
keeps everything from the right table and matches as much as it can from the left tableinner_join
only keeps the observations that are similar between the two tablesfull_join
keeps all observations from both tables.
Let’s play with these CO2 emmissions data just to illustrate:
## read in the data. (same URL as yesterday, just with co2.csv instead of gapminder.csv)
co2 <- read_csv("https://raw.githubusercontent.com/OHI-Science/data-science-training/master/data/co2.csv")
## explore
co2 %>% head()
co2 %>% dim() # 12
## create new variable that is only 2007 data
gap_2007 <- gapminder %>%
filter(year == 2007)
gap_2007 %>% dim() # 142
## left_join gap_2007 to co2
lj <- left_join(gap_2007, co2, by = "country")
## explore
lj %>% dim() #142
lj %>% summary() # lots of NAs in the co2_2017 columm
lj %>% View()
## right_join gap_2007 and co2
rj <- right_join(gap_2007, co2, by = "country")
## explore
rj %>% dim() # 12
rj %>% summary()
rj %>% View()
That’s all we’re going to talk about today with joining, but there are more ways to think about and join your data. Check out the Relational Data Chapter in R for Data Science
7.3.1 load tidyverse
(which has tidyr
inside)
First load tidyr
in an R chunk. You already have installed the tidyverse, so you should be able to just load it like this (using the comment so you can run install.packages("tidyverse")
easily if need be):
library(tidyverse) # install.packages("tidyverse")
7.4 tidyr
basics
Remember, from the dplyr
section, that tidy data means all rows are an observation and all columns are variables.
It’s important to recognize what shape your data is in, and work towards tidying it up.
Let’s take a look at some examples.
Data in the wide format each row is often a site/subject/patient and you have multiple observation variables containing the same type of data. These can be either repeated observations over time, or observation of multiple variables (or a mix of both). Data input may be simpler or some other applications may prefer the wide format. However, many of R
’s functions have been designed assuming you have long format data.
A simple example of data in a wide format is the AirPassengers
dataset which provides information on monthly airline passenger numbers from 1949-1960. You’ll notice that each row is a single year and the columns are each month Jan - Dec.
AirPassengers
## Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
## 1949 112 118 132 129 121 135 148 148 136 119 104 118
## 1950 115 126 141 135 125 149 170 170 158 133 114 140
## 1951 145 150 178 163 172 178 199 199 184 162 146 166
## 1952 171 180 193 181 183 218 230 242 209 191 172 194
## 1953 196 196 236 235 229 243 264 272 237 211 180 201
## 1954 204 188 235 227 234 264 302 293 259 229 203 229
## 1955 242 233 267 269 270 315 364 347 312 274 237 278
## 1956 284 277 317 313 318 374 413 405 355 306 271 306
## 1957 315 301 356 348 355 422 465 467 404 347 305 336
## 1958 340 318 362 348 363 435 491 505 404 359 310 337
## 1959 360 342 406 396 420 472 548 559 463 407 362 405
## 1960 417 391 419 461 472 535 622 606 508 461 390 432
long format is the tidy data we are after, where:
- each column is a variable
- each row is an observation
In the long format, you usually have 1 column for the observed variable and the other columns are ID variables. The mpg
dataset is an example of a long dataset with each row representing a single car and each column representing a variable of that car such as manufacturer
and year
.
mpg
## # A tibble: 234 x 11
## manufacturer model displ year cyl trans drv cty hwy
## <chr> <chr> <dbl> <int> <int> <chr> <chr> <int> <int>
## 1 audi a4 1.8 1999 4 auto(l5) f 18 29
## 2 audi a4 1.8 1999 4 manual(m5) f 21 29
## 3 audi a4 2.0 2008 4 manual(m6) f 20 31
## 4 audi a4 2.0 2008 4 auto(av) f 21 30
## 5 audi a4 2.8 1999 6 auto(l5) f 16 26
## 6 audi a4 2.8 1999 6 manual(m5) f 18 26
## 7 audi a4 3.1 2008 6 auto(av) f 18 27
## 8 audi a4 quattro 1.8 1999 4 manual(m5) 4 18 26
## 9 audi a4 quattro 1.8 1999 4 auto(l5) 4 16 25
## 10 audi a4 quattro 2.0 2008 4 manual(m6) 4 20 28
## # ... with 224 more rows, and 2 more variables: fl <chr>, class <chr>
These different data formats mainly affect readability. For humans, the wide format is often more intuitive since we can often see more of the data on the screen due to it’s shape. However, the long format is more machine readable and is closer to the formatting of databases. The ID variables in our dataframes are similar to the fields in a database and observed variables are like the database values.
Note: Generally, mathematical operations are better in long format, although some plotting functions actually work better with wide format.
Often, data must be reshaped for it to become tidy data. What does that mean? There are four main verbs we’ll use, which are essentially pairs of opposites:
- turn columns into rows (
gather()
), - turn rows into columns (
spread()
), - turn a character column into multiple columns (
separate()
), - turn multiple character columns into a single column (
unite()
)
7.5 Explore gapminder dataset.
Yesterday we started off with the gapminder data in a format that was already tidy. But what if it weren’t? Let’s look at a different version of those data.
The data are on GitHub. Navigate there by going to:
github.com > ohi-science > data-science-training > data > gapminder_wide.csv
or by copy-pasting this in the browser: https://github.com/OHI-Science/data-science-training/blob/master/data/gapminder_wide.csv
Have a look at the data.

You can see there are a lot more columns than the version we looked at before. This format is pretty common, because it can be a lot more intuitive to enter data in this way.
Sometimes, as with the gapminder dataset, we have multiple types of observed data. It is somewhere in between the purely ‘long’ and ‘wide’ data formats:
- 3 “ID variables” (
continent
,country
,year
) - 3 “Observation variables” (
pop
,lifeExp
,gdpPercap
).
It’s pretty common to have data in this intermediate format in most cases despite not having ALL observations in 1 column, since all 3 observation variables have different units. But we can play with switching it to long format and wide to show what that means (i.e. long would be 4 ID variables and 1 observation variable).
But we want it to be in a tidy way so that we can work with it more easily. So here we go.
You use spread()
and gather()
to transform or reshape data between wide to long formats.
7.6 gather()
data from wide to long format
Read in the data from GitHub. Remember, you need to click on the ‘Raw’ button first so you can read it directly. Let’s also read in the gapminder data from yesterday so that we can use it to compare later on.
## wide format
gap_wide <- readr::read_csv('https://raw.githubusercontent.com/OHI-Science/data-science-training/master/data/gapminder_wide.csv')
## yesterday's format (intermediate)
gapminder <- readr::read_csv('https://raw.githubusercontent.com/OHI-Science/data-science-training/master/data/gapminder.csv')
Let’s have a look:
head(gap_wide)
str(gap_wide)
While wide format is nice for data entry, it’s not nice for calculations. Some of the columns are a mix of variable (e.g. “gdpPercap”) and data (“1952”). What if you were asked for the mean population after 1990 in Algeria? Possible, but ugly. But we know it doesn’t need to be so ugly. Let’s tidy it back to the format we’ve been using.
Question: let’s talk this through together. If we’re trying to turn the
gap_wide
format intogapminder
format, what structure does it have that we like? And what do we want to change?
- We like the continent and country columns. We won’t want to change those.
- For long format, we’d want just 1 column identifying the variable name (
tidyr
calls this a ‘key’), and 1 column for the data (tidyr
calls this the ’value’). - For intermediate format, we’d want 3 columns for
gdpPercap
,lifeExp
, andpop
. - We would like year as a separate column.
Let’s get it to long format. We’ll have to do this in 2 steps. The first step is to take all of those column names (e.g. lifeExp_1970
) and make them a variable in a new column, and transfer the values into another column. Let’s learn by doing:
Let’s have a look at gather()
’s help:
?gather
Question: What is our key-value pair?
We need to name two new variables in the key-value pair, one for the key, one for the value. It can be hard to wrap your mind around this, so let’s give it a try. Let’s name them obstype_year
and obs_values
.
Here’s the start of what we’ll do:
gap_long <- gap_wide %>%
gather(key = obstype_year,
value = obs_values)
Although we were already planning to inspect our work, let’s definitely do it now:
str(gap_long)
head(gap_long)
tail(gap_long)
We have reshaped our dataframe but this new format isn’t necessarily what we wanted.
What went wrong? Notice that it didn’t know that we wanted to keep continent
and country
untouched; we need to give it more information about which columns we want reshaped. We can do this in several ways.
One way is to identify the columns is by name. Listing them explicitly can be a good approach if there are just a few. But in our case we have 30 columns. I’m not going to list them out here since there is way too much potential for error if I tried to list gdpPercap_1952
, gdpPercap_1957
, gdpPercap_1962
and so on. But we could use some of dplyr
’s awesome helper functions — because we expect that there is a better way to do this!
gap_long <- gap_wide %>%
gather(key = obstype_year,
value = obs_values,
dplyr::starts_with('pop'),
dplyr::starts_with('lifeExp'),
dplyr::starts_with('gdpPercap')) #here i'm listing all the columns to use in gather
str(gap_long)
head(gap_long)
tail(gap_long)
Success! And there is another way that is nice to use if your columns don’t follow such a structured pattern: you can exclude the columns you don’t want.
gap_long <- gap_wide %>%
gather(key = obstype_year,
value = obs_values,
-continent, -country)
str(gap_long)
head(gap_long)
tail(gap_long)
To recap:
Inside gather()
we first name the new column for the new ID variable (obstype_year
), the name for the new amalgamated observation variable (obs_value
), then the names of the old observation variable. We could have typed out all the observation variables, but as in the select()
function (see dplyr
lesson), we can use the starts_with()
argument to select all variables that starts with the desired character string. Gather also allows the alternative syntax of using the -
symbol to identify which variables are not to be gathered (i.e. ID variables).
OK, but we’re not done yet. obstype_year
actually contains two pieces of information, the observation type (pop
,lifeExp
, or gdpPercap
) and the year
. We can use the separate()
function to split the character strings into multiple variables.
?separate
–> the main arguments are separate(data, col, into, sep ...)
. So we need to specify which column we want separated, name the new columns that we want to create, and specify what we want it to separate by. Since the obstype_year
variable has observation types and years separated by a _
, we’ll use that.
gap_long <- gap_wide %>%
gather(key = obstype_year,
value = obs_values,
-continent, -country) %>%
separate(obstype_year,
into = c('obs_type','year'),
sep = "_",
convert = T) #this ensures that the year column is an integer rather than a character
No warning messages…still we inspect:
str(gap_long)
head(gap_long)
tail(gap_long)
Excellent. This is long format: every row is a unique observation. Yay!
7.7 Plot long format data
The long format is the preferred format for plotting with ggplot2
. Let’s look at an example by plotting just Canada’s life expectency.
canada_df <- gap_long %>%
filter(obs_type == "lifeExp",
country == "Canada")
ggplot(canada_df, aes(x = year, y = obs_values)) +
geom_line()
We can also look at all countries in the Americas:
life_df <- gap_long %>%
filter(obs_type == "lifeExp",
continent == "Americas")
ggplot(life_df, aes(x = year, y = obs_values, color = country)) +
geom_line()
7.7 Exercise
- Using
gap_long
, calculate and plot the the mean life expectancy for each continent over time from 1982 to 2007. Give your plot a title and assign x and y labels. Hint: use thedplyr::group_by()
anddplyr::summarize()
functions.STOP: Knit the R Markdown file and sync to Github (pull, stage, commit, push)
# solution (no peeking!)
gap_long %>%
group_by(continent, obs_type) %>%
summarize(means = mean(obs_values))
cont <- gap_long %>%
filter(obs_type == "lifeExp",
year > 1980) %>%
group_by(continent, year) %>%
summarize(mean_le = mean(obs_values))
ggplot(data = cont, aes(x = year, y = mean_le, color = continent)) +
geom_line() +
labs(title = "Mean life expectancy",
x = "Year",
y = "Age (years)")
## Additional customization
ggplot(data = cont, aes(x = year, y = mean_le, color = continent)) +
geom_line() +
labs(title = "Mean life expectancy",
x = "Year",
y = "Age (years)",
color = "Continent") +
theme_classic() +
scale_fill_brewer(palette = "Blues")
7.8 spread()
The function spread()
is used to transform data from long to intermediate format
Alright! Now just to double-check our work, let’s use the opposite of gather()
to spread our observation variables back to the original format with the aptly named spread()
. You pass spread()
the key and value pair, which is now obs_type
and obs_values
.
gap_normal <- gap_long %>%
spread(obs_type, obs_values)
No warning messages is good…but still let’s check:
dim(gap_normal)
dim(gapminder)
names(gap_normal)
names(gapminder)
Now we’ve got an intermediate dataframe gap_normal
with the same dimensions as the original gapminder
.
7.8 Exercise
Convert “gap_long” all the way back to gap_wide. Hint: you’ll need to create appropriate labels for all our new variables (time*metric combinations) with the opposite of separate:
tidyr::unite()
.Knit the R Markdown file and sync to Github (pull, stage, commit, push)
7.8.1 Answer (no peeking)
head(gap_long) # remember the columns
gap_wide_new <- gap_long %>%
# first unite obs_type and year into a new column called var_names. Separate by _
unite(col = var_names, obs_type, year, sep = "_") %>%
# then spread var_names out by key-value pair.
spread(key = var_names, value = obs_values)
str(gap_wide_new)
7.9 clean up and save your .Rmd
Spend some time cleaning up and saving gapminder-wrangle.Rmd
Restart R. In RStudio, use Session > Restart R. Otherwise, quit R with q()
and re-launch it.
This morning’s .Rmd could look something like this:
## load tidyr (in tidyverse)
library(tidyverse) # install.packages("tidyverse")
## load wide data
gap_wide <- read.csv('https://raw.githubusercontent.com/OHI-Science/data-science-training/master/data/gapminder_wide.csv')
head(gap_wide)
str(gap_wide)
## practice tidyr::gather() wide to long
gap_long <- gap_wide %>%
gather(key = obstype_year,
value = obs_values,
-continent, -country)
# or
gap_long <- gap_wide %>%
gather(key = obstype_year,
value = obs_values,
dplyr::starts_with('pop'),
dplyr::starts_with('lifeExp'),
dplyr::starts_with('gdpPercap'))
## gather() and separate() to create our original gapminder
gap_long <- gap_wide %>%
gather(key = obstype_year,
value = obs_values,
-continent, -country) %>%
separate(obstype_year,
into = c('obs_type','year'),
sep="_")
## practice: can still do calculations in long format
gap_long %>%
group_by(continent, obs_type) %>%
summarize(means = mean(obs_values))
## spread() from normal to wide
gap_normal <- gap_long %>%
spread(obs_type, obs_values) %>%
select(country, continent, year, lifeExp, pop, gdpPercap)
## check that all.equal()
all.equal(gap_normal,gapminder)
## unite() and spread(): convert gap_long to gap_wide
head(gap_long) # remember the columns
gap_wide_new <- gap_long %>%
# first unite obs_type and year into a new column called var_names. Separate by _
unite(col = var_names, obs_type, year, sep = "_") %>%
# then spread var_names out by key-value pair.
spread(key = var_names, value = obs_values)
str(gap_wide_new)
7.9.1 complete()
One of the coolest functions in tidyr
is the function complete()
. Jarrett Byrnes has written up a great blog piece showcasing the utility of this function so I’m going to use that example here.
We’ll start with an example dataframe where the data recorder enters the Abundance of two species of kelp, Saccharina and Agarum in the years 1999, 2000 and 2004.
kelpdf <- data.frame(
Year = c(1999, 2000, 2004, 1999, 2004),
Taxon = c("Saccharina", "Saccharina", "Saccharina", "Agarum", "Agarum"),
Abundance = c(4,5,2,1,8)
)
kelpdf
Jarrett points out that Agarum is not listed for the year 2000. Does this mean it wasn’t observed (Abundance = 0) or that it wasn’t recorded (Abundance = NA)? Only the person who recorded the data knows, but let’s assume that the this means the Abundance was 0 for that year.
We can use the complete()
function to make our dataset more complete.
kelpdf %>% complete(Year, Taxon)
This gives us an NA for Agarum in 2000, but we want it to be a 0 instead. We can use the fill
argument to assign the fill value.
kelpdf %>% complete(Year, Taxon, fill = list(Abundance = 0))
Now we have what we want. Let’s assume that all years between 1999 and 2004 that aren’t listed should actually be assigned a value of 0. We can use the full_seq()
function from tidyr
to fill out our dataset with all years 1999-2004 and assign Abundance values of 0 to those years & species for which there was no observation.
kelpdf %>% complete(Year = full_seq(Year, period = 1),
Taxon,
fill = list(Abundance = 0))


welcome on Tiny Sensor web firmware generator. Choose the sensors you implemented on the hardware, provide some identification information for your data, and generate a ready-to-flash optimized firmware.
Go!


Full-stack developer(Python | PHP | Ruby | React)
As a full-stack web developer, I have backend experience as well as using Python/(Django, Flask), Ruby(Ruby on Rails) with strong background in a variety of database platforms such as Mysql, Postgresql, Mongodb, ElasticSearch.
Especially, I am very fluent in database.
As well as a front-end developer, I have experience in React with HTML5, CSS3, Bootstrap, Jquery, Emberjs, Backbonejs and many other js dependency management.
I had successfully completed Maintaining a Microsoft SQL Sever 2008 Database, an official Microsoft Learning Produce as prescrived by Micosoft Corporation in 2010 and received Microsoft Certified Partner accredits.
Also as had completed Interconnecting Cisco Network Devices II, I received Certificate of Completion from Cisco Systems in 21th Oct, 2011.
Development of information systems for housing and communal services
Managing the information technology Department
If you have any project or question, please ping me at any time. Thank you.