1. Basics of Deep Learning¶
In this notebook, we will cover the basics behind Deep Learning. I’m talking about building a brain….
Only kidding. Deep learning is a fascinating new field that has exploded over the last few years. From being used as facial recognition in apps such as Snapchat or challenger banks, to more advanced use cases such as being used in protein-folding.
In this notebook we will:
Explain the building blocks of neural networks
Go over some applications of Deep Learning
1.1. Building blocks of Neural Networks¶
I have no doubt that you have heard/seen how similar neural networks are to….our brains.
1.1.1. The Perceptron¶
The building block of neural networks. The perceptron has a rich history (covered in the background section of this book). The perceptron was created in 1958 by Frank Rosenblatt (I love that name) in Cornell, however, that story is for another day….or section in this book (backgrounds!),
The perceptron is an algorithm that can learn a binary classifier (e.g. is that a cat or dog?). This is known as a threshold function, which maps an input vector x to an output decision \(f(x)\) = output. Here is the formal maths to better explain my verbal fluff:
\( f(x) = { 1 (if: w.x+b > 0), 0 (otherwise) \)
1.1.2. The Artificial neural network¶
Lets take a look at the high level architecture first.
The gif above of a neural network classifying images is one of the best visual ways of understanding how neural networks, work. The neural network is made up of a few key concepts:
An input: this is the data you pass into the network. For example, data relating to a customer (e.g. height, weight etc) or the pixels of an image
An output: this is the prediction of the neural network
A hidden layer: more on this later
Neuron: the network is made up of neurons, that take an input, and give an output
Now, we have a slightly better understanding of what a neuron is. Lets look at a very simple neuron:
From the above image, you can clearly see the three components listed above together.
1.1.3. But Abdi, what is the goal of a neural network?¶
Isn’t it obvious? To me, it definitely was not when I first started to learn about neural networks. Neural networks are beautifully complex to understand, but with enough time and lots of youtube videos, you’ll be able to master this topic.
The goal of a neural network is to make a pretty good guess of something. For example, a phone may have a face unlock feature. The phone probably got you to take a short video/images of yourself in order to set up this security feature, and when it learned your face, you were then able to use it to unlock your phone. This is pretty much what we do with neural networks. We teach it by giving it data, and making sure it gets better at making predictions by adjusting the weights between neurons. More on this soon.
1.2. Gradient Descent Algo¶
One of the best videos on neural networks, by 3Blue1Brown:
His series on Neural networks and Linear algebra are golden sources for learning Deep Learning.
1.2.1. Simple Gradient Descent Implementation¶
with the help from our friends over at Udacity, please view below an implementation of the Gradient Descent Algo. This is a very basic neural network that only has its inputs linked directly to the outputs.
We begin by defining some functions.
import numpy as np
# We will be using a sigmoid activation function
def sigmoid(x):
return 1/(1+np.exp(-x))
# derivation of sigmoid(x) - will be used for backpropagating errors through the network
def sigmoid_prime(x):
return sigmoid(x)*(1-sigmoid(x))
We begin by defining a simple neural network:
two input neurons: x1 and x2
one output neuron: y1
x = np.array([1,5])
y = 0.4
We now define the weights, w1 and w2 for the two input neurons; x1 and x2. Also, we define a learning rate that will help us control our gradient descent step
weights = np.array([-0.2,0.4])
learnrate = 0.5
we now start moving forwards through the network, known as feed forward. We can combine the input vector with the weight vector using numpy’s dot product
# linear combination
# h = x[0]*weights[0] + x[1]*weights[1]
h = np.dot(x, weights)
We now apply our non-linearity, this will provide us with our output.
# apply non-linearity
output = sigmoid(h)
Now that we have our prediction, we are able to determine the error of our neural network. Here, we will use the difference between our actual and predicted.
error = y - output
The goal now is to determine how to change our weights in order to reduce the error above. This is where our good friend gradient descent and the chain rule come into play:
we determine the derivative of our error with respect to our input weights. Hence:
change in weights = \( \frac{d}{dw_{i}} \frac{1}{2}{(y - \hat{y})^2}\)
simplifies to = learning rate * error term * \( x_{i}\)
where:
learning rate = \( n \)
error term = \( (y - \hat{y}) * f'(h) \)
h = \( \sum_{i} W_{i} x_{i} \)
We begin by calculating our f’(h)
# output gradient - derivative of activation function
output_gradient = sigmoid_prime(h)
Now, we can calcualte our error term
error_trm = error * output_gradient
With that, we can update our weights by combining the error term, learning rate and our x
#gradient desc step - updating the weights
dsc_step = [
learnrate * error_trm * x[0],
learnrate * error_trm * x[1]
]
Which leaves…
print(f'Actual: {y}')
print(f'NN output: {output}')
print(f'Error: {error}')
print(f'Weight change: {dsc_step}')
Actual: 0.4
NN output: 0.8581489350995123
Error: -0.45814893509951227
Weight change: [-0.02788508381144715, -0.13942541905723577]
1.2.2. More in depth…¶
Lets now build our own end to end example. we will begin by creating some fake data, followed by implementing our neural network.
x = np.random.rand(200,2)
y = np.random.randint(low=0, high=2, size=(200,1))
no_data_points, no_features = x.shape
def sig(x):
'''Calc for sigmoid'''
return 1 / (1+np.exp(-x))
weights = np.random.normal(scale=1/no_features**.5, size=no_features)
epochs = 1000
learning_rate = 0.5
last_loss = None
for single_data_pass in range(epochs):
# Creating a weight change tracker
change_in_weights = np.zeros(weights.shape)
for x_i, y_i in zip(x, y):
h = np.dot(x_i, weights)
y_hat = sigmoid(h)
error = y_i - y_hat
# error term = error * f'(h)
error_term = error * (y_hat * (1-y_hat))
# now multiply this by the current x & add to our weight update
change_in_weights += (error_term * x_i)
# now update the actual weights
weights += (learning_rate * change_in_weights / no_data_points)
# print the loss every 100th pass
if single_data_pass % (epochs/10) == 0:
# use current weights in NN to determine outputs
output = sigmoid(np.dot(x_i,weights))
# find the loss
loss = np.mean((output-y_i)**2)
#
if last_loss and last_loss < loss:
print(f'Train loss: {loss}, WARNING - Loss is inscreasing')
else:
print(f'Training loss: {loss}')
last_loss = loss
Training loss: 0.20879328109199216
Train loss: 0.24423433937166253, WARNING - Loss is inscreasing
Train loss: 0.2535774687986323, WARNING - Loss is inscreasing
Train loss: 0.25806549907726695, WARNING - Loss is inscreasing
Train loss: 0.2612052379782168, WARNING - Loss is inscreasing
Train loss: 0.2636314466602466, WARNING - Loss is inscreasing
Train loss: 0.265540779202265, WARNING - Loss is inscreasing
Train loss: 0.2670478508460244, WARNING - Loss is inscreasing
Train loss: 0.2682382618486228, WARNING - Loss is inscreasing
Train loss: 0.2691790525966896, WARNING - Loss is inscreasing
1.3. Multilayer NN¶
Now, lets build upon our neural network, but this time, we have a hidden layer.
Lets first see how to build the network to make predictions.
X = np.random.randn(4)
weights_input_to_hidden = np.random.normal(0, scale=0.1, size=(4, 3))
weights_hidden_to_output = np.random.normal(0, scale=0.1, size=(3, 2))
sum_input = np.dot(X, weights_input_to_hidden)
h = sigmoid(sum_input)
sum_h = np.dot(h, weights_hidden_to_output)
y_pred = sigmoid(sum_h)
1.4. Backpropa what?¶
Ok, so now, how do we refine our weights? Well, this is where backpropagation comes in. After feeding our data forwards through the network, using feed forward, we propagate our errors backwards, making use of things such as the chain rule.
Lets do an implementation.
# we have three input nodes
x = np.array([0.5, 0.2, -0.3])
# one output node
y = 0.7
learnrate = 0.5
# 2 nodes in hidden layer
weights_input_hidden = np.array(
[
[0.5, -0.6], [0.1, -0.2], [0.1, 0.7]
]
)
weights_hidden_output = np.array([
0.1,-0.3
])
# feeding data forwards through the network
hidden_layer_input = np.dot(x, weights_input_hidden)
hidden_layer_output = sigmoid(hidden_layer_input)
#---
output_layer_input = np.dot(hidden_layer_output, weights_hidden_output)
y_hat = sigmoid(output_layer_input)
# backward propagate the errors to tune the weights
# 1. calculate errors
error = y - y_hat
output_node_error_term = error * (y_hat * (1-y_hat))
#----
hidden_node_error_term = weights_hidden_output * output_node_error_term *(hidden_layer_output * (1-hidden_layer_output))
# 2. calculate weight changes
delta_w_output_node = learnrate * output_node_error_term * hidden_layer_output
#-----
delta_w_hidden_node = learnrate * hidden_node_error_term * x[:,None]
print(f'Original weights:\n{weights_input_hidden}\n{weights_hidden_output}')
print()
print('Change in weights for hidden layer to output layer:')
print(delta_w_output_node)
print('Change in weights for input layer to hidden layer:')
print(delta_w_hidden_node)
Original weights:
[[ 0.5 -0.6]
[ 0.1 -0.2]
[ 0.1 0.7]]
[ 0.1 -0.3]
Change in weights for hidden layer to output layer:
[0.01492263 0.00975438]
Change in weights for input layer to hidden layer:
[[ 0.00032851 -0.00092784]
[ 0.0001314 -0.00037114]
[-0.00019711 0.0005567 ]]
1.5. Putting it all together¶
features = np.random.rand(200,2)
target = np.random.randint(low=0, high=2, size=(200,1))
def complete_backprop(x,y):
'''Complete implementation of backpropagation'''
n_hidden_units = 2
epochs = 900
learnrate = 0.005
n_records, n_features = features.shape
last_loss = None
w_input_to_hidden = np.random.normal(scale=1/n_features**.5,size=(n_features, n_hidden_units))
w_hidden_to_output = np.random.normal(scale=1/n_features**.5, size=n_hidden_units)
for single_epoch in range(epochs):
delw_input_to_hidden = np.zeros(w_input_to_hidden.shape)
delw_hidden_to_output = np.zeros(w_hidden_to_output.shape)
for x,y in zip(features, target):
# ----------------------
# 1. Feed data forwards
# ----------------------
hidden_layer_input = np.dot(x,w_input_to_hidden)
hidden_layer_output = sigmoid(hidden_layer_input)
output_layer_input = np.dot(hidden_layer_output, w_hidden_to_output)
output_layer_output = sigmoid(output_layer_input)
# ----------------------
# 2. Backpropagate the errors
# ----------------------
# error at output layer
prediction_error = y - output_layer_output
output_error_term = prediction_error * (output_layer_output * (1-output_layer_output))
# error at hidden layer (propagated from output layer)
# scale error from output layer by weights
hidden_layer_error = np.multiply(output_error_term, w_hidden_to_output)
hidden_error_term = hidden_layer_error * (hidden_layer_output * (1-hidden_layer_output))
# ----------------------
# 3. Find change of weights for each data point
# ----------------------
delw_hidden_to_output += output_error_term * hidden_layer_output
delw_input_to_hidden += hidden_error_term * x[:,None]
# Now update the actual weights
w_hidden_to_output += learnrate * delw_hidden_to_output / n_records
w_input_to_hidden += learnrate * delw_input_to_hidden / n_records
# Printing out the mean square error on the training set
if single_epoch % (epochs / 10) == 0:
hidden_output = sigmoid(np.dot(x, w_input_to_hidden))
out = sigmoid(np.dot(hidden_output,
w_hidden_to_output))
loss = np.mean((out - target) ** 2)
if last_loss and last_loss < loss:
print("Train loss: ", loss, " WARNING - Loss Increasing")
else:
print("Train loss: ", loss)
last_loss = loss
complete_backprop(features,target)
Train loss: 0.2516119393037275
Train loss: 0.2515507327943288
Train loss: 0.2514911890398006
Train loss: 0.2514332619591989
Train loss: 0.251376906713852
Train loss: 0.25132207967725206
Train loss: 0.2512687384054615
Train loss: 0.2512168416080365
Train loss: 0.2511663491194731
Train loss: 0.25111722187117796


@(locations: List[Location]) @main("Assignment List") {
There must be at least one location before you can add an assignment. Use the button below to add a location.
} else {Click on a location to manage assignments for that location.
| Location | Street | City | |
|---|---|---|---|
| @location.getName | @location.getAddress1 | @location.getCity |


WITNESS LIST
Intergovernmental Relations
October 23, 2014 1:00 PM
Study the structure of county government for efficiency/effectiveness.
ON:
Allison, Jim General Counsel (County Judges and Commissioners Association of Texas),
Austin, TX
Betancourt, David County Treasurers Assn. Of Texas, President (County Treasurers' Assn.
Of Texas), Brownsville, TX
Bethke, Jim Executive Director (also providing written testimony) (Texas Indigent
Defense Commission), Austin, TX
Crider, Rachael (Self; Galveston County Tax Office), Galveston, Tx, TX
Dion, Edward County Auditor (also providing written testimony) (Self; Texas Association
of County Auditors), El Paso, TX
Flores, David County Auditor (Texas Association Of County Auditors), Georgetown, TX
Johnson, Cheryl Co tax assessor collector (also providing written testimony) (Self;
Galveston County Tax Office), Galveston, TX
Joseph, Zenobia Ms. (also providing written testimony) (Self), Austin, TX
Kepple, Rob (Texas district and county attorneys association), Austin, TX
Lee, Donald Executive Director (Texas Conference of Urban Counties), Austin, TX
Ortega Carter, Dolores Travis County Treasurer (also providing written testimony) (Self;
County Treasurers Association of Tx), Austin, TX
Renken, David County auditor (Tx association of county Auditors), New braunfels, TX
Swift, Sheryl (Galveston County Tax Office), Galveston, TX
Velasquez, Sally Frio County Governmental Affairs Consultant (also providing written
testimony) (Frio County Commissioners Court), Pearsall, TX
Registering, but not testifying:
ON:
Austin, David County auditor (Self; Texas assoc of county auditors), Odessa, TX
Brown, Cindy Denton County Treasurer (Self; County Treasurers Association of Texas),
Denton, TX
Cantu, Priscilla County Treasurer (Self; County Treasurer's Association of Texas),
Kingsville, TX
Conner, Katie President, Texas Association of County Auditors (Self; Texas Association
of County Auditors), Bryan, TX
Cradduck, Nathan County Auditor (Self; Texas Association of County Auditors), San
angelo, TX
Dollahan, Rick County auditor (Self; Texas assoc of county auditors), Seminole, TX
Garcia, Norma G Hidalgo County Treasurer (Self; County Treasurers' Assn.), Edinburg,
TX
Garner Dolezal, Krissa Budget and Financial Reporting Supervisor (Montgomery County,
Texas), Conroe, TX
Kiley, Julie First Assistant County Auditor (Self; Texas Association of County Auditors),
Georgetown, TX
Klein, Kristen County Auditor (Self; Texas association of county auditors), Seguin, TX
Martin, Phyllis Montgomery County Auditor (Self; Texas Association of County
Auditors), Conroe, TX
WITNESS LIST
Intergovernmental Relations
October 23, 2014 1:00 PM
Yeatts, Susan County Auditor (Texas Association of County Auditors), San Antonio, TX,
TX


A histogram is a graphical representation of the distribution of data. It is an estimate of the probability distribution of a continuous variable.
A scatter plot, scatterplot is a type of diagram using Cartesian coordinates to display values for two variables for a set of data.
A 3D scatter plot, scatterplot is a type of mathematical diagram using Cartesian coordinates to display values for three variables for a set of data.