Recognizing multi-digit numbers in photographs captured at street level is an important component of modern-day map making. A classic example of a corpus of such street level photographs is Google’s Street View imagery comprised of hundreds of millions of geo-located 360 degree panoramic images. The ability to automatically transcribe an address number from a geo-located patch of pixels and associate the transcribed number with a known street address helps pinpoint, with a high degree of accuracy, the location of the building it represents.
More broadly, recognizing numbers in photographs is a problem of interest to the optical character recognition community. While OCR on constrained domains like document processing is well studied, arbitrary multi-character text recognition in photographs is still highly challenging. This difficulty arises due to the wide variability in the visual appearance of text in the wild on account of a large range of fonts, colors, styles, orientations, and character arrangements. The recognition problem is further complicated by environmental factors such as lighting, shadows, specularities, and occlusions as well as by image acquisition factors such as resolution, motion, and focus blurs.
In this project we will use dataset with images centred around a single digit (many of the images do contain some distractors at the sides). Although we are taking a sample of the data which is simpler, it is more complex than MNIST because of the distractors.
The Street View House Numbers (SVHN) Dataset
SVHN is a real-world image dataset for developing machine learning and object recognition algorithms with minimal requirement on data formatting but comes from a significantly harder, unsolved, real world problem (recognizing digits and numbers in natural scene images). SVHN is obtained from house numbers in Google Street View images.
Acknowledgement for the datasets
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, Andrew Y. Ng Reading Digits in Natural Images with Unsupervised Feature Learning NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011. PDF http://ufldl.stanford.edu/housenumbers as the URL for this site when necessary.
Objective of the project is to learn how to implement a simple image classification pipeline based on a deep neural network.
# Mounting Google Drive
from google.colab import drive
drive.mount('/content/drive')
# Setting the current working directory
import os; os.chdir('drive/My Drive/Great Learning/Neural Network')
# Imports
import pandas as pd, numpy as np, matplotlib.pyplot as plt, seaborn as sns, h5py
import matplotlib.style as style; style.use('fivethirtyeight')
%matplotlib inline
# Metrics and preprocessing
from sklearn.metrics import classification_report, accuracy_score, confusion_matrix, precision_score, recall_score, f1_score, precision_recall_curve, auc
from sklearn.model_selection import train_test_split
from sklearn import preprocessing
# TF and Keras
from tensorflow.keras.layers import Dense, Dropout, BatchNormalization
from tensorflow.keras.layers import Activation, Dense
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.models import Sequential
from tensorflow.keras import optimizers
# Checking if GPU is found
import tensorflow as tf
device_name = tf.test.gpu_device_name()
if device_name != '/device:GPU:0':
raise SystemError('GPU device not found')
print('Found GPU at: {}'.format(device_name))
tf.reset_default_graph()
tf.set_random_seed(42)
!ls '/content/drive/My Drive/Great Learning/Neural Network'
# Read the h5 file
h5_SVH = h5py.File('SVHN_single_grey1.h5', 'r')
# Load the training, validation and test sets
X_train = h5_SVH['X_train'][:]
y_train_o = h5_SVH['y_train'][:]
X_val = h5_SVH['X_val'][:]
y_val_o = h5_SVH['y_val'][:]
X_test = h5_SVH['X_test'][:]
y_test_o = h5_SVH['y_test'][:]
# Close this file
h5_SVH.close()
print('Training set', X_train.shape, y_train_o.shape)
print('Validation set', X_val.shape, y_val_o.shape)
print('Test set', X_test.shape, y_test_o.shape)
print('\n')
print('Unique labels in y_train:', np.unique(y_train_o))
print('Unique labels in y_val:', np.unique(y_val_o))
print('Unique labels in y_test:', np.unique(y_test_o))
# Visualizing first 10 images in the dataset and their labels
plt.figure(figsize = (15, 4.5))
for i in range(10):
plt.subplot(1, 10, i+1)
plt.imshow(X_train[i].reshape((32, 32)),cmap = plt.cm.binary)
plt.axis('off')
plt.subplots_adjust(wspace = -0.1, hspace = -0.1)
plt.show()
print('Label for each of the above image: %s' % (y_train_o[0 : 10]))
print('Checking first image and label in training set'); print('--'*40)
plt.imshow(X_train[0], cmap = plt.cm.binary)
plt.show()
print('Label:', y_train_o[0])
print('Checking first image and label in validation set'); print('--'*40)
plt.imshow(X_val[0], cmap = plt.cm.binary)
plt.show()
print('Label:', y_val_o[0])
print('Checking first image and label in test set'); print('--'*40)
plt.imshow(X_test[0], cmap = plt.cm.binary)
plt.show()
print('Label:', y_test_o[0])
print('Reshaping X data: (n, 32, 32) => (n, 1024)'); print('--'*40)
X_train = X_train.reshape((X_train.shape[0], -1))
X_val = X_val.reshape((X_val.shape[0], -1))
X_test = X_test.reshape((X_test.shape[0], -1))
print('Making sure that the values are float so that we can get decimal points after division'); print('--'*40)
X_train = X_train.astype('float32')
X_val = X_val.astype('float32')
X_test = X_test.astype('float32')
print('Normalizing the RGB codes by dividing it to the max RGB value'); print('--'*40)
X_train /= 255
X_val /= 255
X_test /= 255
print('Converting y data into categorical (one-hot encoding)'); print('--'*40)
y_train = to_categorical(y_train_o)
y_val = to_categorical(y_val_o)
y_test = to_categorical(y_test_o)
print('X_train shape:', X_train.shape)
print('X_val shape:', X_val.shape)
print('X_test shape:', X_test.shape)
print('\n')
print('y_train shape:', y_train.shape)
print('y_val shape:', y_val.shape)
print('y_test shape:', y_test.shape)
print('\n')
print('Number of images in X_train', X_train.shape[0])
print('Number of images in X_val', X_val.shape[0])
print('Number of images in X_test', X_test.shape[0])
class Linear():
def __init__(self, in_size, out_size):
self.W = np.random.randn(in_size, out_size) * 0.01
self.b = np.zeros((1, out_size))
self.params = [self.W, self.b]
self.gradW = None
self.gradB = None
self.gradInput = None
def forward(self, X):
self.X = X
self.output = np.dot(X, self.W) + self.b
return self.output
def backward(self, nextgrad):
self.gradW = np.dot(self.X.T, nextgrad)
self.gradB = np.sum(nextgrad, axis=0)
self.gradInput = np.dot(nextgrad, self.W.T)
return self.gradInput, [self.gradW, self.gradB]
class ReLU():
def __init__(self):
self.params = []
self.gradInput = None
def forward(self, X):
self.output = np.maximum(X, 0)
return self.output
def backward(self, nextgrad):
self.gradInput = nextgrad.copy()
self.gradInput[self.output <=0] = 0
return self.gradInput, []
def softmax(x):
exp_x = np.exp(x - np.max(x, axis=1, keepdims=True))
return exp_x / np.sum(exp_x, axis=1, keepdims=True)
class CrossEntropy:
def forward(self, X, y):
self.m = y.shape[0]
self.p = softmax(X)
cross_entropy = -np.log(self.p[range(self.m), y]+1e-16)
loss = np.sum(cross_entropy) / self.m
return loss
def backward(self, X, y):
y_idx = y.argmax()
grad = softmax(X)
grad[range(self.m), y] -= 1
grad /= self.m
return grad
class NN():
def __init__(self, lossfunc = CrossEntropy(), mode = 'train'):
self.params = []
self.layers = []
self.loss_func = lossfunc
self.grads = []
self.mode = mode
def add_layer(self, layer):
self.layers.append(layer)
self.params.append(layer.params)
def forward(self, X):
for layer in self.layers:
X = layer.forward(X)
return X
def backward(self, nextgrad):
self.clear_grad_param()
for layer in reversed(self.layers):
nextgrad, grad = layer.backward(nextgrad)
self.grads.append(grad)
return self.grads
def train_step(self, X, y):
out = self.forward(X)
loss = self.loss_func.forward(out,y)
nextgrad = self.loss_func.backward(out,y)
grads = self.backward(nextgrad)
return loss, grads
def predict(self, X):
X = self.forward(X)
p = softmax(X)
return np.argmax(p, axis=1)
def predict_scores(self, X):
X = self.forward(X)
p = softmax(X)
return p
def clear_grad_param(self):
self.grads = []
def update(velocity, params, grads, learning_rate=0.01, mu=0.9):
for v, p, g, in zip(velocity, params, reversed(grads)):
for i in range(len(g)):
v[i] = (mu * v[i]) - (learning_rate * g[i])
p[i] += v[i]
def minibatch(X, y, minibatch_size):
n = X.shape[0]
minibatches = []
permutation = np.random.permutation(X.shape[0])
X = X[permutation]
y = y[permutation]
for i in range(0, n , minibatch_size):
X_batch = X[i:i + minibatch_size, :]
y_batch = y[i:i + minibatch_size, ]
minibatches.append((X_batch, y_batch))
return minibatches
def train(net, X_train, y_train, minibatch_size, epoch, learning_rate, mu = 0.9, X_val = None, y_val = None, Lambda = 0, verb = True):
val_loss_epoch = []
minibatches = minibatch(X_train, y_train, minibatch_size)
minibatches_val = minibatch(X_val, y_val, minibatch_size)
for i in range(epoch):
loss_batch = []
val_loss_batch = []
velocity = []
for param_layer in net.params:
p = [np.zeros_like(param) for param in list(param_layer)]
velocity.append(p)
# iterate over mini batches
for X_mini, y_mini in minibatches:
loss, grads = net.train_step(X_mini, y_mini)
loss_batch.append(loss)
update(velocity, net.params, grads, learning_rate=learning_rate, mu=mu)
for X_mini_val, y_mini_val in minibatches_val:
val_loss, _ = net.train_step(X_mini, y_mini)
val_loss_batch.append(val_loss)
# accuracy of model at end of epoch after all mini batch updates
m_train = X_train.shape[0]
m_val = X_val.shape[0]
y_train_pred = []
y_val_pred = []
y_train1 = []
y_vall = []
for ii in range(0, m_train, minibatch_size):
X_tr = X_train[ii:ii + minibatch_size, : ]
y_tr = y_train[ii:ii + minibatch_size,]
y_train1 = np.append(y_train1, y_tr)
y_train_pred = np.append(y_train_pred, net.predict(X_tr))
for ii in range(0, m_val, minibatch_size):
X_va = X_val[ii:ii + minibatch_size, : ]
y_va = y_val[ii:ii + minibatch_size,]
y_vall = np.append(y_vall, y_va)
y_val_pred = np.append(y_val_pred, net.predict(X_va))
train_acc = check_accuracy(y_train1, y_train_pred)
val_acc = check_accuracy(y_vall, y_val_pred)
## weights
w = np.array(net.params[0][0])
## adding regularization to cost
mean_train_loss = (sum(loss_batch) / float(len(loss_batch)))
mean_val_loss = sum(val_loss_batch) / float(len(val_loss_batch))
val_loss_epoch.append(mean_val_loss)
if verb:
if i%50==0:
print("Epoch {3}/{4}: Loss = {0} | Training Accuracy = {1}".format(mean_train_loss, train_acc, val_acc, i, epoch))
return net, val_acc
def check_accuracy(y_true, y_pred):
return np.mean(y_pred == y_true)
# Invoking the model
## input size
input_dim = X_train.shape[1]
def train_and_test_loop(iterations, lr, Lambda, verb = True):
## hyperparameters
iterations = iterations
learning_rate = lr
hidden_nodes1 = 10
output_nodes = 10
## define neural net
nn = NN()
nn.add_layer(Linear(input_dim, hidden_nodes1))
nn, val_acc = train(nn, X_train, y_train_o, minibatch_size = 200, epoch = iterations, learning_rate = learning_rate,\
X_val = X_test, y_val = y_test_o, Lambda = Lambda, verb = verb)
return val_acc
lr = 0.00001
Lambda = 0
train_and_test_loop(1, lr, Lambda)
lr = 0.00001
Lambda = 1e3
train_and_test_loop(1, lr, Lambda)
X_train_subset = X_train[0:20]
y_train_subset = y_train_o[0:20]
X_train = X_train_subset
y_train_o = y_train_subset
X_train.shape, y_train_o.shape
So, set a small learning rate and turn regularization off In the code below:
%time
lr = 0.001
Lambda = 0
train_and_test_loop(5000, lr, Lambda)
h5_SVH = h5py.File('SVHN_single_grey1.h5', 'r')
# Load the training, validation and test sets
X_train = h5_SVH['X_train'][:]
y_train_o = h5_SVH['y_train'][:]
X_val = h5_SVH['X_val'][:]
y_val_o = h5_SVH['y_val'][:]
X_test = h5_SVH['X_test'][:]
y_test_o = h5_SVH['y_test'][:]
print('Reshaping X data: (n, 32, 32) => (n, 1024)'); print('--'*40)
X_train = X_train.reshape((X_train.shape[0], -1))
X_val = X_val.reshape((X_val.shape[0], -1))
X_test = X_test.reshape((X_test.shape[0], -1))
print('Making sure that the values are float so that we can get decimal points after division'); print('--'*40)
X_train = X_train.astype('float32')
X_val = X_val.astype('float32')
X_test = X_test.astype('float32')
print('Normalizing the RGB codes by dividing it to the max RGB value'); print('--'*40)
X_train /= 255
X_val /= 255
X_test /= 255
print('Converting y data into categorical (one-hot encoding)'); print('--'*40)
y_train = to_categorical(y_train_o)
y_val = to_categorical(y_val_o)
y_test = to_categorical(y_test_o)
lr = 1e-7
Lambda = 1e-7
train_and_test_loop(500, lr, Lambda)
lr = 0.001
Lambda = 1e-7
train_and_test_loop(500, lr, Lambda)
import math
for k in range(1, 10):
lr = math.pow(10, np.random.uniform(-3.0, -2.0))
Lambda = math.pow(10, np.random.uniform(-5, 2))
best_acc = train_and_test_loop(100, lr, Lambda, False)
print("Try {0}/{1}: Best_val_acc: {2}, lr: {3}, Lambda: {4}\n".format(k, 10, best_acc, lr, Lambda))
print('NN model with sigmoid activations'); print('--'*40)
# Initialize the neural network classifier
model1 = Sequential()
# Input Layer - adding input layer and activation functions sigmoid
model1.add(Dense(128, input_shape = (1024, )))
# Adding activation function
model1.add(Activation('sigmoid'))
#Hidden Layer 1 - adding first hidden layer
model1.add(Dense(64))
# Adding activation function
model1.add(Activation('sigmoid'))
# Output Layer - adding output layer which is of 10 nodes (digits)
model1.add(Dense(10))
# Adding activation function - softmax for multiclass classification
model1.add(Activation('softmax'))
model1.summary()
# compiling the neural network classifier, sgd optimizer
sgd = optimizers.SGD(lr = 0.01)
model1.compile(optimizer = sgd, loss = 'categorical_crossentropy', metrics = ['accuracy'])
# Fitting the neural network for training
history = model1.fit(X_train, y_train, validation_data = (X_val, y_val), batch_size = 200, epochs = 100, verbose = 1)
print('Evaluate NN model with sigmoid activations'); print('--'*40)
results1 = model1.evaluate(X_val, y_val)
print('Validation accuracy: {}'.format(round(results1[1]*100, 2), '%'))
print('NN model with sigmoid activations - changing learning rate'); print('--'*40)
# compiling the neural network classifier, sgd optimizer
sgd = optimizers.SGD(lr = 0.001)
model1.compile(optimizer = sgd, loss = 'categorical_crossentropy', metrics = ['accuracy'])
# Fitting the neural network for training
history = model1.fit(X_train, y_train, validation_data = (X_val, y_val), batch_size = 200, epochs = 100, verbose = 1)
print('Evaluate NN model with sigmoid activations - changing learning rate'); print('--'*40)
results1 = model1.evaluate(X_val, y_val)
print('Validation accuracy: {}'.format(round(results1[1]*100, 2), '%'))
%time
print('NN model with relu activations and sgd optimizers'); print('--'*40)
# Initialize the neural network classifier
model2 = Sequential()
# Input Layer - adding input layer and activation functions relu
model2.add(Dense(128, input_shape = (1024, )))
# Adding activation function
model2.add(Activation('relu'))
#Hidden Layer 1 - adding first hidden layer
model2.add(Dense(64))
# Adding activation function
model2.add(Activation('relu'))
# Output Layer - adding output layer which is of 10 nodes (digits)
model2.add(Dense(10))
# Adding activation function - softmax for multiclass classification
model2.add(Activation('softmax'))
model2.summary()
# compiling the neural network classifier, sgd optimizer
sgd = optimizers.SGD(lr = 0.01)
model2.compile(optimizer = sgd, loss = 'categorical_crossentropy', metrics = ['accuracy'])
# Fitting the neural network for training
history = model2.fit(X_train, y_train, validation_data = (X_val, y_val), batch_size = 200, epochs = 100, verbose = 1)
print('Evaluate NN model with relu activations'); print('--'*40)
results2 = model2.evaluate(X_val, y_val)
print('Validation accuracy: {}'.format(round(results2[1]*100, 2), '%'))
%time
print('NN model with relu activations and sgd optimizers - changing learning rate'); print('--'*40)
# compiling the neural network classifier, sgd optimizer
sgd = optimizers.SGD(lr = 0.001)
model2.compile(optimizer = sgd, loss = 'categorical_crossentropy', metrics = ['accuracy'])
# Fitting the neural network for training
history = model2.fit(X_train, y_train, validation_data = (X_val, y_val), batch_size = 200, epochs = 100, verbose = 1)
print('Evaluate NN model with relu activations'); print('--'*40)
results2 = model2.evaluate(X_val, y_val)
print('Validation accuracy: {}'.format(round(results2[1]*100, 2), '%'))
%time
print('NN model with relu activations and adam optimizer'); print('--'*40)
# compiling the neural network classifier, adam optimizer
adam = optimizers.Adam(lr = 0.01)
model2.compile(optimizer = adam, loss = 'categorical_crossentropy', metrics = ['accuracy'])
# Fitting the neural network for training
history = model2.fit(X_train, y_train, validation_data = (X_val, y_val), batch_size = 200, epochs = 100, verbose = 1)
print('Evaluate NN model with relu activations'); print('--'*40)
results2 = model2.evaluate(X_val, y_val)
print('Validation accuracy: {}'.format(round(results2[1]*100, 2), '%'))
%time
print('NN model with relu activations and adam optimizer'); print('--'*40)
# compiling the neural network classifier, adam optimizer
adam = optimizers.Adam(lr = 0.001)
model2.compile(optimizer = adam, loss = 'categorical_crossentropy', metrics = ['accuracy'])
# Fitting the neural network for training
history = model2.fit(X_train, y_train, validation_data = (X_val, y_val), batch_size = 200, epochs = 100, verbose = 1)
print('Evaluate NN model with relu activations'); print('--'*40)
results2 = model2.evaluate(X_val, y_val)
print('Validation accuracy: {}'.format(round(results2[1]*100, 2), '%'))
print('NN model with relu activations and changing number of activators'); print('--'*40)
# Initialize the neural network classifier
model3 = Sequential()
# Input Layer - adding input layer and activation functions relu
model3.add(Dense(256, input_shape = (1024, )))
# Adding activation function
model3.add(Activation('relu'))
#Hidden Layer 1 - adding first hidden layer
model3.add(Dense(128))
# Adding activation function
model3.add(Activation('relu'))
#Hidden Layer 2 - Adding second hidden layer
model3.add(Dense(64))
# Adding activation function
model3.add(Activation('relu'))
# Output Layer - adding output layer which is of 10 nodes (digits)
model3.add(Dense(10))
# Adding activation function - softmax for multiclass classification
model3.add(Activation('softmax'))
model3.summary()
# compiling the neural network classifier, sgd optimizer
sgd = optimizers.SGD(lr = 0.01)
model3.compile(optimizer = sgd, loss = 'categorical_crossentropy', metrics = ['accuracy'])
# Fitting the neural network for training
history = model3.fit(X_train, y_train, validation_data = (X_val, y_val), batch_size = 200, epochs = 100, verbose = 1)
print('Evaluate NN model with relu activations and changing the number of activators'); print('--'*40)
results3 = model3.evaluate(X_val, y_val)
print('Validation accuracy: {}'.format(round(results3[1]*100, 2), '%'))
# compiling the neural network classifier, adam optimizer
adam = optimizers.Adam(lr = 0.001)
model3.compile(optimizer = adam, loss = 'categorical_crossentropy', metrics = ['accuracy'])
# Fitting the neural network for training
history = model3.fit(X_train, y_train, validation_data = (X_val, y_val), batch_size = 200, epochs = 100, verbose = 1)
print('Evaluate NN model with relu activations and changing the number of activators'); print('--'*40)
results3 = model3.evaluate(X_val, y_val)
print('Validation accuracy: {}'.format(round(results3[1]*100, 2), '%'))
Changing weight initialization scheme can significantly improve training of the model by preventing vanishing gradient problem up to some degree.
print('NN model with weight initializers'); print('--'*40)
# Initialize the neural network classifier
model4 = Sequential()
# Input Layer - adding input layer and activation functions relu and weight initializer
model4.add(Dense(256, input_shape = (1024, ), kernel_initializer = 'he_normal'))
# Adding activation function
model4.add(Activation('relu'))
#Hidden Layer 1 - adding first hidden layer
model4.add(Dense(128, kernel_initializer = 'he_normal', bias_initializer = 'he_uniform'))
# Adding activation function
model4.add(Activation('relu'))
#Hidden Layer 2 - adding second hidden layer
model4.add(Dense(64, kernel_initializer = 'he_normal', bias_initializer = 'he_uniform'))
# Adding activation function
model4.add(Activation('relu'))
#Hidden Layer 3 - adding third hidden layer
model4.add(Dense(32, kernel_initializer = 'he_normal', bias_initializer = 'he_uniform'))
# Adding activation function
model4.add(Activation('relu'))
# Output Layer - adding output layer which is of 10 nodes (digits)
model4.add(Dense(10, kernel_initializer = 'he_normal', bias_initializer = 'he_uniform'))
# Adding activation function
model4.add(Activation('softmax'))
model4.summary()
# compiling the neural network classifier, sgd optimizer
sgd = optimizers.SGD(lr = 0.01)
# Adding activation function - softmax for multiclass classification
model4.compile(optimizer = sgd, loss = 'categorical_crossentropy', metrics = ['accuracy'])
# Fitting the neural network for training
history = model4.fit(X_train, y_train, validation_data = (X_val, y_val), batch_size = 200, epochs = 100, verbose = 1)
print('NN with weight initializers'); print('--'*40)
results4 = model4.evaluate(X_val, y_val)
print('Validation accuracy: {}'.format(round(results4[1]*100, 2), '%'))
# compiling the neural network classifier, adam optimizer
adam = optimizers.Adam(lr = 0.001)
# Adding activation function - softmax for multiclass classification
model4.compile(optimizer = adam, loss = 'categorical_crossentropy', metrics = ['accuracy'])
# Fitting the neural network for training
history = model4.fit(X_train, y_train, validation_data = (X_val, y_val), batch_size = 200, epochs = 100, verbose = 1)
print('NN with weight initializers'); print('--'*40)
results4 = model4.evaluate(X_val, y_val)
print('Validation accuracy: {}'.format(round(results4[1]*100, 2), '%'))
Batch Normalization, one of the methods to prevent the "internal covariance shift" problem, has proven to be highly effective. Normalize each mini-batch before nonlinearity.
print('NN model with batch normalization'); print('--'*40)
# Initialize the neural network classifier
model5 = Sequential()
# Input Layer - adding input layer and activation functions relu and weight initializer
model5.add(Dense(256, input_shape = (1024, ), kernel_initializer = 'he_normal'))
# Adding batch normalization
model5.add(BatchNormalization())
# Adding activation function
model5.add(Activation('relu'))
#Hidden Layer 1 - adding first hidden layer
model5.add(Dense(128, kernel_initializer = 'he_normal', bias_initializer = 'he_uniform'))
# Adding batch normalization
model5.add(BatchNormalization())
# Adding activation function
model5.add(Activation('relu'))
#Hidden Layer 2 - adding second hidden layer
model5.add(Dense(64, kernel_initializer = 'he_normal', bias_initializer = 'he_uniform'))
# Adding batch normalization
model5.add(BatchNormalization())
# Adding activation function
model5.add(Activation('relu'))
#Hidden Layer 3 - adding third hidden layer
model5.add(Dense(32, kernel_initializer = 'he_normal', bias_initializer = 'he_uniform'))
# Adding batch normalization
model5.add(BatchNormalization())
# Adding activation function
model5.add(Activation('relu'))
# Output Layer - adding output layer which is of 10 nodes (digits)
model5.add(Dense(10, kernel_initializer = 'he_normal', bias_initializer = 'he_uniform'))
# Adding activation function
model5.add(Activation('softmax'))
model5.summary()
# compiling the neural network classifier, sgd optimizer
sgd = optimizers.SGD(lr = 0.01)
# Adding activation function - softmax for multiclass classification
model5.compile(optimizer = sgd, loss = 'categorical_crossentropy', metrics = ['accuracy'])
# Fitting the neural network for training
history = model5.fit(X_train, y_train, validation_data = (X_val, y_val), batch_size = 200, epochs = 100, verbose = 1)
print('NN with batch normalization'); print('--'*40)
results5 = model5.evaluate(X_val, y_val)
print('Validation accuracy: {}'.format(round(results5[1]*100, 2), '%'))
# compiling the neural network classifier, adam optimizer
adam = optimizers.Adam(lr = 0.001)
# Adding activation function - softmax for multiclass classification
model5.compile(optimizer = adam, loss = 'categorical_crossentropy', metrics = ['accuracy'])
# Fitting the neural network for training
history = model5.fit(X_train, y_train, validation_data = (X_val, y_val), batch_size = 200, epochs = 100, verbose = 1)
print('NN with batch normalization'); print('--'*40)
results5 = model5.evaluate(X_val, y_val)
print('Validation accuracy: {}'.format(round(results5[1]*100, 2), '%'))
print('NN model with dropout - sgd optimizer'); print('--'*40)
# Initialize the neural network classifier
model6 = Sequential()
# Input Layer - adding input layer and activation functions relu and weight initializer
model6.add(Dense(512, input_shape = (1024, ), kernel_initializer = 'he_normal'))
# Adding batch normalization
model6.add(BatchNormalization())
# Adding activation function
model6.add(Activation('relu'))
# Adding dropout layer
model6.add(Dropout(0.2))
#Hidden Layer 1 - adding first hidden layer
model6.add(Dense(256, kernel_initializer = 'he_normal', bias_initializer = 'he_uniform'))
# Adding batch normalization
model6.add(BatchNormalization())
# Adding activation function
model6.add(Activation('relu'))
# Adding dropout layer
model6.add(Dropout(0.2))
#Hidden Layer 2 - adding second hidden layer
model6.add(Dense(128, kernel_initializer = 'he_normal', bias_initializer = 'he_uniform'))
# Adding batch normalization
model6.add(BatchNormalization())
# Adding activation function
model6.add(Activation('relu'))
# Adding dropout layer
model6.add(Dropout(0.2))
#Hidden Layer 3 - adding third hidden layer
model6.add(Dense(64, kernel_initializer = 'he_normal', bias_initializer = 'he_uniform'))
# Adding batch normalization
model6.add(BatchNormalization())
# Adding activation function
model6.add(Activation('relu'))
# Adding dropout layer
model6.add(Dropout(0.2))
#Hidden Layer 4 - adding fourth hidden layer
model6.add(Dense(32, kernel_initializer = 'he_normal', bias_initializer = 'he_uniform'))
# Adding batch normalization
model6.add(BatchNormalization())
# Adding activation function
model6.add(Activation('relu'))
# Adding dropout layer
model6.add(Dropout(0.2))
# Output Layer - adding output layer which is of 10 nodes (digits)
model6.add(Dense(10, kernel_initializer = 'he_normal',bias_initializer = 'he_uniform'))
# Adding activation function
model6.add(Activation('softmax'))
model6.summary()
# compiling the neural network classifier, sgd optimizer
sgd = optimizers.SGD(lr = 0.01)
model6.compile(optimizer = sgd, loss = 'categorical_crossentropy', metrics = ['accuracy'])
# Adding activation function - softmax for multiclass classification
history = model6.fit(X_train, y_train, validation_data = (X_val, y_val), batch_size = 200, epochs = 100, verbose = 1)
print('NN model with dropout - sgd optimizer'); print('--'*40)
results6 = model6.evaluate(X_val, y_val)
print('Validation accuracy: {}'.format(round(results6[1]*100, 2), '%'))
# compiling the neural network classifier, adam optimizer
adam = optimizers.Adam(lr = 0.001)
model6.compile(optimizer = adam, loss = 'categorical_crossentropy', metrics = ['accuracy'])
# Adding activation function - softmax for multiclass classification
history = model6.fit(X_train, y_train, validation_data = (X_val, y_val), batch_size = 200, epochs = 100, verbose = 1)
print('NN model with dropout - adam optimizer'); print('--'*40)
results6 = model6.evaluate(X_val, y_val)
print('Validation accuracy: {}'.format(round(results6[1]*100, 2), '%'))
print('NN model with relu activations and changing number of activators'); print('--'*40)
# Initialize the neural network classifier
model3 = Sequential()
# Input Layer - adding input layer and activation functions relu
model3.add(Dense(256, input_shape = (1024, )))
# Adding activation function
model3.add(Activation('relu'))
#Hidden Layer 1 - adding first hidden layer
model3.add(Dense(128))
# Adding activation function
model3.add(Activation('relu'))
#Hidden Layer 2 - Adding second hidden layer
model3.add(Dense(64))
# Adding activation function
model3.add(Activation('relu'))
# Output Layer - adding output layer which is of 10 nodes (digits)
model3.add(Dense(10))
# Adding activation function - softmax for multiclass classification
model3.add(Activation('softmax'))
# compiling the neural network classifier, adam optimizer
adam = optimizers.Adam(lr = 0.001)
model3.compile(optimizer = adam, loss = 'categorical_crossentropy', metrics = ['accuracy'])
# Fitting the neural network for training
history = model3.fit(X_train, y_train, validation_data = (X_val, y_val), batch_size = 200, epochs = 100, verbose = 1)
print('NN with batch normalization'); print('--'*40)
results3 = model3.evaluate(X_val, y_val)
print('Validation accuracy: {}'.format(round(results3[1]*100, 2), '%'))
print('Testing the model on test dataset')
predictions = model3.predict_classes(X_test)
score = model3.evaluate(X_test, y_test)
print('Test loss :', score[0])
print('Test accuracy :', score[1])
print('Classification Report'); print('--'*40)
print(classification_report(y_test_o, predictions))
print('Visualizing the confusion matrix')
plt.figure(figsize = (15, 7.2))
sns.heatmap(confusion_matrix(y_test_o, predictions), annot = True)
# summarize history for accuracy
plt.plot(history.history['acc'])
plt.plot(history.history['val_acc'])
plt.title('Model3 Accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['Train', 'Validation'], loc = 'upper left')
plt.show()
# summarize history for loss
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('Model3 Loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['Train', 'Validation'], loc = 'upper left')
plt.show()
model3.predict_classes(X_test)[5]
#Showing the image
plt.imshow(X_test[20].reshape(32, 32), cmap = 'gray')
model3.predict_classes(X_test)[20]
plt.imshow(X_test[10].reshape(32, 32), cmap = 'gray')
model3.predict_classes(X_test)[10]
Evaluated the accuracy using two methods i.e. baby sitting the NN and NN through API. Followed all the required steps starting with loading the datasets to performing hyperparameter optimization and running a finer search by using a finer range. Explored different options in optimizers, number of activators, learning rate and activation methods in NN through API. Found that baby sitting process achieved the best accuracy of 21% using hyper parameter optimization. It might have been further improved but that's the trade off vs time taken to run the script. NN through API method achieved best accuracy of 90% on validation set. Also printed the classification report, visualized the confusion matrix and summarized history for accuracy and loss.


Ansible is available as a package in most OS's.
See: Installation
Make a folder for your Ansible files.
See: Getting started
[sites]
<IP_ADDRESS>
<IP_ADDRESS>
<IP_ADDRESS>
<IP_ADDRESS>
This is a list of hosts you want to manage, grouped into groups. (Hint:
try using localhost ansible_connection=local
to deploy to
your local machine.)
See: Intro to Inventory
- hosts: <IP_ADDRESS>
user: root
tasks:
- name: install nginx
apt: pkg=nginx state=present
- name: start nginx every bootup
service: name=nginx state=started enabled=yes
- name: do something in the shell
shell: echo hello > /tmp/abc.txt
- name: install bundler
gem: name=bundler state=latest
See: Intro to Playbooks
~/setup$ ls
hosts
playbook.yml
~/setup$ ansible-playbook -i hosts playbook.yml
PLAY [all] ********************************************************************
GATHERING FACTS ***************************************************************
ok: [<IP_ADDRESS>]
TASK: [install nginx] *********************************************************
ok: [<IP_ADDRESS>]
TASK: start nginx every bootup] ***********************************************
ok: [<IP_ADDRESS>]
...


ITest
Information about a test.
Fields
date: Date - date when test will happen
dayOfWeek: string - day of the week when test will happen
subject: string - subject
type: string - type
issuedBy: string - name of teacher that issued the test
scope: string - scope
id: number - entry id


Create New Cluster Template
Brief overview of various middleware frameworks
Cluster Template Frameworks:
Some info here relating to HRCondor and WQ
+ New Cluster Template
ADD FORM AND INFO HERE regarding SPARK
{%extends "loginbase.html"%} {%block title%}Create Environment{%endblock%} {%block body%} {%include 'messages.html'%}
Some info here relating to HRCondor and WQ
+ New Cluster Template


PRIZZI'S HONOR A film review by Walter Frith Copyright 1997 Walter Frith
Director John Huston who died in shortly after making this film was a master of story telling. There was always a hint of conservative ambience detailing his films and for a man who started in the 1940's and made movies for over four decades, he certainly was the right man for the very dark and off beat mob comedy, 'Prizzi's Honor' (1985). It ranks as a superlative look at a fictional New York City crime family and is based on Richard Condon's best selling novel.
The film opens in charismatic style at a wedding and each of the main characters is visually profiled. There is the ever unpredictable Jack Nicholson as a hit man for the Prizzi family and the other main characters in the film include Kathleen Turner as the woman he would fall in love with and who turns out to be more than he bargained for. Others are William Hickey as the family boss who personifies the devil in his gripping visual style of acting. Lee Richardson and Robert Loggia as his sons, John Randolph as Nicholson's father and best friend to Hickey and Anjelica Huston (daughter of director Huston and winner of the Best Supporting Actress Oscar for this film) as the Prizzi family's mafia princess and a former lover of Nicholson's in the film.
Nicholson is love struck at the wedding as he glimpses Turner for the first time and tries to track her down and they eventually consumate their relationship by marrying and it turns out that while Nicholson is a slow witted hit man, she's a quick witted hit woman! This of course causes problems in their relationship as she becomes embroiled in deceiving the Prizzi family and is offered a contract to rub out Nicholson. The outrageous twists that this film offers are too numerous to mention and 'Prizzi's Honor' also serves as a cleverly inter woven and well plotted film which never loses its focus in the important moments and has one more goody up its sleeve just when you think things are ready to settle down.
If you haven't seen 'Prizzi's Honor' it's real treat and is perfect for home video viewing and is recommended for adult viewing for its mature theme. It's one of Nicholson's most pleasantly menacing performances.
The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


▅ ▆ ▇ █ HAPPY fes_name █ ▇ ▆ ▅
Happy Birthday
Wish You A Very Happy
Birthday Anmol.May You Live 100Years.


# (c) 2012-2014, Michael DeHaan<EMAIL_ADDRESS>#
# This file is part of Ansible
#
# Ansible is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# Ansible is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with Ansible. If not, see <http://www.gnu.org/licenses/>.
from __future__ import (absolute_import, division, print_function)
__metaclass__ = type
import sys
from ansible import constants as C
ANSIBLE_COLOR=True
if C.ANSIBLE_NOCOLOR:
ANSIBLE_COLOR=False
elif not hasattr(sys.stdout, 'isatty') or not sys.stdout.isatty():
ANSIBLE_COLOR=False
else:
try:
import curses
curses.setupterm()
if curses.tigetnum('colors') < 0:
ANSIBLE_COLOR=False
except ImportError:
# curses library was not found
pass
except curses.error:
# curses returns an error (e.g. could not find terminal)
ANSIBLE_COLOR=False
if C.ANSIBLE_FORCE_COLOR:
ANSIBLE_COLOR=True
# --- begin "pretty"
#
# pretty - A miniature library that provides a Python print and stdout
# wrapper that makes colored terminal text easier to use (e.g. without
# having to mess around with ANSI escape sequences). This code is public
# domain - there is no license except that you must leave this header.
#
# Copyright (C) 2008 Brian Nez <thedude at bri1 dot com>
#
# http://nezzen.net/2008/06/23/colored-text-in-python-using-ansi-escape-sequences/
codeCodes = {
'black': u'0;30', 'bright gray': u'0;37',
'blue': u'0;34', 'white': u'1;37',
'green': u'0;32', 'bright blue': u'1;34',
'cyan': u'0;36', 'bright green': u'1;32',
'red': u'0;31', 'bright cyan': u'1;36',
'purple': u'0;35', 'bright red': u'1;31',
'yellow': u'0;33', 'bright purple': u'1;35',
'dark gray': u'1;30', 'bright yellow': u'1;33',
'magenta': u'0;35', 'bright magenta': u'1;35',
'normal': u'0' ,
}
[docs]def stringc(text, color):
"""String in color."""
if ANSIBLE_COLOR:
return u"\033[%sm%s\033[0m" % (codeCodes[color], text)
else:
return text
# --- end "pretty"
[docs]def colorize(lead, num, color):
""" Print 'lead' = 'num' in 'color' """
if num != 0 and ANSIBLE_COLOR and color is not None:
return u"%s%s%-15s" % (stringc(lead, color), stringc("=", color), stringc(str(num), color))
else:
return u"%s=%-4s" % (lead, str(num))
[docs]def hostcolor(host, stats, color=True):
if ANSIBLE_COLOR and color:
if stats['failures'] != 0 or stats['unreachable'] != 0:
return u"%-37s" % stringc(host, 'red')
elif stats['changed'] != 0:
return u"%-37s" % stringc(host, 'yellow')
else:
return u"%-37s" % stringc(host, 'green')
return u"%-26s" % host


PrimeOne for Figma uses the latest powerful features like components, variants, auto layout, styles and interactive components. It'll always follow the best practices.
Save countless hours on every project with a carefully designed system that uses Prime UI Suite components. Start producing design results in no time.
PrimeOne is designed based on Lara Blue Light and Lara Blue Dark themes. Easily change the themes of your designs using Figma's Swap Library feature.
Empower yourself with unprecedented control over your designs. Tokens Studio integration unlocks a whole new level of flexibility, allowing you to create and manage design tokens seamlessly.
You'll be able to download two Figma files for light and dark themes.
UI Kit license is perpetual so requires one time payment, not subscription based.
Yes, your license allows you to sell your projects that utilize the UI Kit implementations.
There is no limit, you are able to use UI Kit in multiple projects for multiple clients.
Yes, after the purchase, please contact us so we can transfer the license to your client.
Yes, contractors are also able to use the UI Kit within your company.
No, enterprise license is per company so each subsidiary company needs to purchase a separate license.
All updates will be totally free of charge for existing customers for an unlimited period.
Support is provided by PrimeTek via a dedicated forum channel monitored by PrimeTek support staff.
Support service at the forum does not have a time limit.
Due to the license, it is not possible to use the UI Kit in an open source project where the design files are publicly available.


The Best New Deals + Exclusive Offers from RetailMeNot!
Up to 80% off Gold Star clearance, plus save an extra 20% off sitewide when you use your Kohl’s card! Expires 3/13/13
More top offers include:
Starbucks Store: Free Tall Hazelnut or Caramel Macchiatto
Macy’s: Up to 20% Off Sitewide + Free Shipping on $99 or More
Overstock: 15% Off All Furniture Exclusive to RetailMeNot.com!
Sierra Trading Post: 25% Off $75 or More
& many more at www.retailmenot.com
Copyright 2013 WhaleShark Media. 301 Congress Avenue, Suite 700, Austin, Texas 78701
To opt-out please reply to this email with UNSUBSCRIBE in the subject line.


Before Change
// when x < 0 and otherwise.
// Hence sample uniformly from [-6, 0] and
// [0, 6] and provide a warm-start.
X1 = (-6 + 6 * rng.rand(5, 1)).tolist()
X2 = (6 * rng.rand(5, 1)).tolist()
X0 = X1 + X2
yield (check_minimize, minimizer, bench2, -5,
[(-6.0, 6.0)], 0.05, 100, 0, X0)
yield (check_minimize, minimizer, bench3, -0.9,
After Change
// if all the randomly sampled points are one side of the
// the minimum, since for a decision tree any point greater than
// max(sampled_points) would give a constant value.
X0 = [[-5.6], [-5.8], [5.8], [5.6]]
yield (check_minimize, minimizer, bench2, -5,
[(-6.0, 6.0)], 0.1, 100, 10, X0)
yield (check_minimize, minimizer, bench3, -0.9,


Before Change
self.pocket = pocket
self.hand = hand
pocket_score, pocket_cards, pocket_result = oc.calc_score(pocket)
board_score, board_cards, board_result = oc.calc_score(board)
hand_score, hand_cards, hand_result = oc.calc_score(hand)
"""print("----------pocket-------")
print(pocket_score)
print(pocket_cards)
print(pocket_result)
print("----------board-------")
print(board_score)
print(board_cards)
print(board_result)
print("----------hand-------")
print(hand_score)
print(hand_cards)
print(hand_result)"""
print("---------CALCULATING OUTS--------")
self.calculate_outs(pocket_score, pocket_cards, pocket_result, board_score, board_cards, board_result, hand_score,
hand_cards, hand_result, hand)
def calculate_outs(self, pocket_score, pocket_cards, pocket_result, board_score, board_cards, board_result,
hand_score,
hand_cards, hand_result, hand):
After Change
oc.board_score, oc.board_cards, oc.board_result = oc.calc_score(oc.board)
oc.hand_score, oc.hand_cards, oc.hand_result = oc.calc_score(oc.hand)
outs = self.calculate_outs(oc)
return outs
def calculate_outs(self, oc):


${pageProperties.jcr:title}
properties.jcr:title: ${properties.jcr:title}
currentPage.pageTitle: ${currentPage.pageTitle}
currentPage.path: ${currentPage.path}
currentStyle: ${currentStyle}
http://adobeaemclub.com/guide-for-working-with-templates-and-call-in-sightly-aem-6-1/
This is my first Template. Value of x = ${x}, y= ${y}, z= ${z}


“I tweeted the photo to my FaceBook, but now I don’t see it on my friend’s list. How do I get it to show up?.”
Wait… you did what, and you’re trying to do what??
When it comes to anything technical I think the most important part of asking a good question is understanding the terminology and using it correctly so that you are asking the question that you mean to ask. It’s common for someone who doesn’t understand the terminology to use it incorrectly and that can make a question impossible to answer because it doesn’t make sense in the first place.
While working as a software consultant I spent a lot of time interpreting my client’s questions to try to understand what they were really asking. In the example above, something simple like “How to post a photo to my friend’s FaceBook timeline?” would have been a much better question, and is likely to get a better response from someone with the technical knowledge needed to accomplish the goal.
When it comes to coding questions in particular, usually the situation is going to look something like this:
All three of these components need to be included in a good question. If the question were a statement instead it would look like this: “I’m trying to accomplish A (the goal) by doing B (the attempt), and I’m getting C (the error).” To translate that into a question you might phrase it something like this, “Using B to accomplish A, why am I getting C instead of A?”
Another thing to consider when phrasing and framing a question is: Would I be able to answer this if I knew the answer? Did you include all of the necessary details and information that someone will need to know to give you an answer? This is especially true if you’ve already tried the most logical first step(s) to fix an issue, and those didn’t work. You need to include that additional info of what you’ve already tried. In general you might include a snippet of code to help show what you’re doing, or include the specific error message text or a screenshot of the error that you’re getting.
I have a handful of utility software applications that I developed for QuickBooks desktop and sell online. When I do support for my utility software apps, I always ask the user for a screenshot of the error message. That’s so much more helpful than “it doesn’t work” or “I get an error”. When I developed the applications, I handled the most common errors, so the error text tells me exactly what’s going on. This makes troubleshooting and answering user questions much easier.
If you do post something in a forum or other location expecting an answer, it’s common to need to provide more information or details, so be prepared to provide that if you want to get the best answer to your question. If you want to get very comfortable with asking questions, and increase your technical knowledge at the same time, try answering other people’s questions.


{% set no_robots = True %} {% extends "layout.html" %} {% block title %}Generate Unique IDs{% endblock %} {% block content %}
This page will simply generate a list of unique IDs that are guaranteed to be unique within the context of this application and are therefore suitable for assignment to samples and organisms. The prefix is optional and the uniqueness of the IDs does not depend on this prefix, so there is no risk of ID conflicts in cases where the suffix needs to be modified after ID generation.
{% if unique_ids %}| Unique IDs |
|---|
| {{ unique_id }} |


[#list lotterys as lottery]
[/#list]
[#else]
${lottery.lotteryName}
${lottery.endDate}
没有抽奖活动
[#include "/includes/header.html" /]
${lottery.lotteryName}
${lottery.endDate}
没有抽奖活动


Why Education for In the conference on Environment and Development held by the UN in Rio de Janeiro 1992 it was agreed by all countries to form a sustainable development for 21st century named Agenda 21. In this it was stated that education is essential to further a sustainable development and to improve peoples ability to solve environmental and development problems. Ten years later, in 2002, a UN World Summit Meeting took place in Johannesburg in which status was stated over the progress of sustainability of each country. In the meeting it was realized that education is one of the most important tools in the work for a sustainable world. Subsequently UN adopted a resolution to speed up the process of Education for Sustainable Development covering the next decade 2005-2014. The UN organisation, UNESCO, is the coordinator of the project for the period 2005-2014. The responsibility for carrying out the work lies within the UNESCO National Commission in each country. UNESCO’S vision for the decade:
|
|
|---|


Frequently Asked Questions¶
How does Scrapy compare to BeautifulSoup or lxml?¶
BeautifulSoup and lxml are libraries for parsing HTML and XML. Scrapy is an application framework for writing web spiders that crawl web sites and extract data from them.
Scrapy provides a built-in mechanism for extracting data (called selectors) but you can easily use BeautifulSoup (or lxml) instead, if you feel more comfortable working with them. After all, they’re just parsing libraries which can be imported and used from any Python code.
In other words, comparing BeautifulSoup (or lxml) to Scrapy is like comparing jinja2 to Django.
Can I use Scrapy with BeautifulSoup?¶
Yes, you can.
As mentioned above, BeautifulSoup can be used
for parsing HTML responses in Scrapy callbacks.
You just have to feed the response’s body into a BeautifulSoup
object
and extract whatever data you need from it.
Here’s an example spider using BeautifulSoup API, with lxml
as the HTML parser:
from bs4 import BeautifulSoup
import scrapy
class ExampleSpider(scrapy.Spider):
name = "example"
allowed_domains = ["example.com"]
start_urls = (
'http://www.example.com/',
)
def parse(self, response):
# use lxml to get decent HTML parsing speed
soup = BeautifulSoup(response.text, 'lxml')
yield {
"url": response.url,
"title": soup.h1.string
}
Note
BeautifulSoup
supports several HTML/XML parsers.
See BeautifulSoup’s official documentation on which ones are available.
What Python versions does Scrapy support?¶
Scrapy is supported under Python 2.7 and Python 3.4+ under CPython (default Python implementation) and PyPy (starting with PyPy 5.9). Python 2.6 support was dropped starting at Scrapy 0.20. Python 3 support was added in Scrapy 1.1. PyPy support was added in Scrapy 1.4, PyPy3 support was added in Scrapy 1.5.
Note
For Python 3 support on Windows, it is recommended to use Anaconda/Miniconda as outlined in the installation guide.
Did Scrapy “steal” X from Django?¶
Probably, but we don’t like that word. We think Django is a great open source project and an example to follow, so we’ve used it as an inspiration for Scrapy.
We believe that, if something is already done well, there’s no need to reinvent it. This concept, besides being one of the foundations for open source and free software, not only applies to software but also to documentation, procedures, policies, etc. So, instead of going through each problem ourselves, we choose to copy ideas from those projects that have already solved them properly, and focus on the real problems we need to solve.
We’d be proud if Scrapy serves as an inspiration for other projects. Feel free to steal from us!
Does Scrapy work with HTTP proxies?¶
Yes. Support for HTTP proxies is provided (since Scrapy 0.8) through the HTTP
Proxy downloader middleware. See
HttpProxyMiddleware
.
How can I scrape an item with attributes in different pages?¶
Scrapy crashes with: ImportError: No module named win32api¶
You need to install pywin32 because of this Twisted bug.
How can I simulate a user login in my spider?¶
See Using FormRequest.from_response() to simulate a user login.
Does Scrapy crawl in breadth-first or depth-first order?¶
By default, Scrapy uses a LIFO queue for storing pending requests, which basically means that it crawls in DFO order. This order is more convenient in most cases. If you do want to crawl in true BFO order, you can do it by setting the following settings:
DEPTH_PRIORITY = 1
SCHEDULER_DISK_QUEUE = 'scrapy.squeues.PickleFifoDiskQueue'
SCHEDULER_MEMORY_QUEUE = 'scrapy.squeues.FifoMemoryQueue'
My Scrapy crawler has memory leaks. What can I do?¶
Also, Python has a builtin memory leak issue which is described in Leaks without leaks.
How can I make Scrapy consume less memory?¶
See previous question.
Can I use Basic HTTP Authentication in my spiders?¶
Yes, see HttpAuthMiddleware
.
Why does Scrapy download pages in English instead of my native language?¶
Try changing the default Accept-Language request header by overriding the
DEFAULT_REQUEST_HEADERS
setting.
Can I run a spider without creating a project?¶
Yes. You can use the runspider
command. For example, if you have a
spider written in a my_spider.py
file you can run it with:
scrapy runspider my_spider.py
See runspider
command for more info.
I get “Filtered offsite request” messages. How can I fix them?¶
Those messages (logged with DEBUG
level) don’t necessarily mean there is a
problem, so you may not need to fix them.
Those messages are thrown by the Offsite Spider Middleware, which is a spider middleware (enabled by default) whose purpose is to filter out requests to domains outside the ones covered by the spider.
For more info see:
OffsiteMiddleware
.
What is the recommended way to deploy a Scrapy crawler in production?¶
See Deploying Spiders.
Can I use JSON for large exports?¶
It’ll depend on how large your output is. See this warning in JsonItemExporter
documentation.
Can I return (Twisted) deferreds from signal handlers?¶
Some signals support returning deferreds from their handlers, others don’t. See the Built-in signals reference to know which ones.
What does the response status code 999 means?¶
999 is a custom response status code used by Yahoo sites to throttle requests.
Try slowing down the crawling speed by using a download delay of 2
(or
higher) in your spider:
class MySpider(CrawlSpider):
name = 'myspider'
download_delay = 2
# [ ... rest of the spider code ... ]
Or by setting a global download delay in your project with the
DOWNLOAD_DELAY
setting.
Can I call pdb.set_trace()
from my spiders to debug them?¶
Yes, but you can also use the Scrapy shell which allows you to quickly analyze
(and even modify) the response being processed by your spider, which is, quite
often, more useful than plain old pdb.set_trace()
.
For more info see Invoking the shell from spiders to inspect responses.
Simplest way to dump all my scraped items into a JSON/CSV/XML file?¶
To dump into a JSON file:
scrapy crawl myspider -o items.json
To dump into a CSV file:
scrapy crawl myspider -o items.csv
To dump into a XML file:
scrapy crawl myspider -o items.xml
For more information see Feed exports
What’s this huge cryptic __VIEWSTATE
parameter used in some forms?¶
The __VIEWSTATE
parameter is used in sites built with ASP.NET/VB.NET. For
more info on how it works see this page. Also, here’s an example spider
which scrapes one of these sites.
What’s the best way to parse big XML/CSV data feeds?¶
Parsing big feeds with XPath selectors can be problematic since they need to build the DOM of the entire feed in memory, and this can be quite slow and consume a lot of memory.
In order to avoid parsing all the entire feed at once in memory, you can use
the functions xmliter
and csviter
from scrapy.utils.iterators
module. In fact, this is what the feed spiders (see Spiders) use
under the cover.
Does Scrapy manage cookies automatically?¶
Yes, Scrapy receives and keeps track of cookies sent by servers, and sends them back on subsequent requests, like any regular web browser does.
For more info see Requests and Responses and CookiesMiddleware.
How can I see the cookies being sent and received from Scrapy?¶
Enable the COOKIES_DEBUG
setting.
How can I instruct a spider to stop itself?¶
Raise the CloseSpider
exception from a callback. For
more info see: CloseSpider
.
How can I prevent my Scrapy bot from getting banned?¶
Should I use spider arguments or settings to configure my spider?¶
Both spider arguments and settings can be used to configure your spider. There is no strict rule that mandates to use one or the other, but settings are more suited for parameters that, once set, don’t change much, while spider arguments are meant to change more often, even on each spider run and sometimes are required for the spider to run at all (for example, to set the start url of a spider).
To illustrate with an example, assuming you have a spider that needs to log into a site to scrape data, and you only want to scrape data from a certain section of the site (which varies each time). In that case, the credentials to log in would be settings, while the url of the section to scrape would be a spider argument.
I’m scraping a XML document and my XPath selector doesn’t return any items¶
You may need to remove namespaces. See Removing namespaces.


October Sky (1999) Jake Gyllenhaal, Chris Cooper, William Lee Scott, Chris Owen, Chad Lindberg, Natalie Canerday, Laura Dern, Scott Miles, Scott Thomas, Courtney Fendley, "Big Frank" Schuler, Kailie Hollister, Jeff Galpin, O. Winston Link. Screenplay by Lewis Colick, based on the book "Rocket Boys" by Homer H. Hickam Jr. Directed by Joe Johnston. 108 minutes. Rated PG, 4 stars (out of five stars)
Review by Ed Johnson-Ott, NUVO Newsweekly www.nuvo-online.com Archive reviews at http://us.imdb.com/M/reviews_by?Edward+Johnson-ott To receive reviews by e-mail at no charge, send subscription requests to<EMAIL_ADDRESS>Think of "Hoosiers," with a science fair subbing for a basketball game as the grand finale, and you'll have an idea of what to expect from "October Sky," an earnest, inspirational throwback to the pre-ironic days of filmmaking. How refreshing to watch idealism and sentimentality presented onscreen without the usual cheap manipulative tricks so beloved by Hollywood schlockmeisters. Admittedly, "October Sky" is at times a bit too pat, but given the sheer good will engendered by the movie, that flaw is easy to take.
The story, based on Homer J. Hickam Jr.'s memoir, "Rocket Boys," begins on a brisk night in 1957, when the residents of Coalwood, West Virginia watch the Russian satellite Sputnik glide across the starry October sky. Inspired by the sight, and the U.S. efforts to keep up with their Cold War rivals, young Homer Hickam decides to start building rockets of his own, much to the dismay of his father, the mine superintendent for the company town's only industry. Along with a few friends, Homer forms the "Big Creek Missile Agency" and pursues his dream, encouraged by a teacher who believes the boys might have a chance of winning the National Science Fair in Indianapolis and securing the college scholarships necessary for them to escape life in the mines.
The film has two story arcs, both well-traveled but satisfying nonetheless. One is a tale of perseverance, chronicling the boys' dogged efforts to succeed through study, experimentation and hard work. The other eloquently details one young man's quest to be his own person, even while he continues to desperately crave the approval of his father. In a letter to his former high school, the real Homer Hickam stated he wrote his book "for all of us who watched our parents sacrifice in a million ways every day so that we might have a better life. It was written for all of us who observed by deed every day how much our parents loved us but never experienced it through touch or word." If you found that statement corny rather than moving, this may not be the film for you.
"October Sky" captures the look and feel of a tiny '50s coal-mining town so well that the golden-oldies soundtrack seems unnecessary and intrusive. The cast does fine work making the town come to life. Laura Dern is effective as the boy's teacher and Natalie Canerday gives a great performance as Elsie Hickam, a fiercely devoted mother and wife who hates Coalwood. Even while battling for her family, she paints a beachside mural on the kitchen wall, a small act of defiance against this town of soot, hacking coughs and lives of resignation.
As Homer, Jake Gyllenhaal nicely straddles the line between wide-eyed boy and fiery young man. He makes Homer a 20th Century Huck Finn, with a personality born of his mother's dreams and his father's backbone. Chris Cooper, in the film's most crucial role, plays John Hickam with grit and intensity, tempered by a hint of sadness in his eyes. As a superintendent, husband and father, John Hickam is firm and not always fair, but he truly believes himself a reasonable man operating in the best interests of those around him. While you will certainly understand Homer's frustration with his father, you'll also realize why he cherishes him so.
After watching "October Sky," I bought and read "Rocket Boys," partially because I wanted to spend more time with the Hickams, but also to check the accuracy of the film, which seemed almost too slick to be true. Indeed, the movie makes some changes. The six rocket boys in the "Big Creek Missile Agency" are condensed into four characters. Homer's mother was considerably more assertive, and his father quietly more helpful, than portrayed in the film. An incident with the law which casts a pall over the boys for a goodly portion of the movie was actually resolved in one afternoon. And as for one of the few images in the film that looked contrived, a shot of a defeated young Homer gazing upwards at Sputnik while a mine elevator lowers him into the bowels of the earth, well, let's just call it poetic license and leave it at that.
The liberties taken by the filmmakers didn't really bother me, though. The movie is true to the essence of the book and, most importantly, it does not dishonor Homer's parents, John and Elsie Hickam. That matters, because it shows that the people behind "October Sky" adhere to the values they espouse. As with "Hoosiers," "October Sky" may move some facts around and throw in a fictional subplot, but it maintains the integrity of the true story behind the celluloid.
© 1999 Ed Johnson-Ott
The review above was posted to the
rec.arts.movies.reviews newsgroup (de.rec.film.kritiken for German reviews).
The Internet Movie Database accepts no responsibility for the contents of the
review and has no editorial control. Unless stated otherwise, the copyright
belongs to the author.
Please direct comments/criticisms of the review to relevant newsgroups.
Broken URLs inthe reviews are the responsibility of the author.
The formatting of the review is likely to differ from the original due
to ASCII to HTML conversion.
Related links: index of all rec.arts.movies.reviews reviews


Under the Creative Commons (BY-NC-SA) 3.0 License - Header image by Marlis Funk - Powered by Octopress


Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 1 29 Feb 2016
Everyone should be done with Assignment 3 now
Milestone grades will go out soon
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 2 29 Feb 2016
Spatial Transformer
Segmentation
Soft Attention
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 3 29 Feb 2016
Videos
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 4 29 Feb 2016
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 5 29 Feb 2016
Dense trajectories and motion boundary descriptors for action recognition Wang et al., 2013
Action Recognition with Improved Trajectories
Wang and Schmid, 2013
(code available!)
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 6 29 Feb 2016
Dense trajectories and motion boundary descriptors for action recognition Wang et al., 2013
|
|
|
|||
|
|||||
|
|||||
|
|||||
|
|||||
|
|
7 |
|
||
Wang et al., 2013
detected feature points
[J. Shi and C. Tomasi, “Good features to track,” CVPR 1994] [Ivan Laptev 2005]
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 8 29 Feb 2016
Wang et al., 2013
track each keypoint using optical flow.
[G. Farnebäck, “Two-frame motion estimation based on polynomial expansion,” 2003]
[T. Brox and J. Malik, “Large displacement optical flow: Descriptor matching in variational motion estimation,” 2011]
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 9 29 Feb 2016
Wang et al., 2013
| Extract features in the local coordinate |
|
|
|
||
| system of each tracklet. | ||
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 10 29 Feb 2016
[Krizhevsky et al. 2012]
Input: 227x227x3 images
First layer (CONV1): 96 11x11 filters applied at stride 4 =>
Output volume [55x55x96]
Q: What if the input is now a small chunk of video? E.g. [227x227x3x15] ?
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 11 29 Feb 2016
[Krizhevsky et al. 2012]
Input: 227x227x3 images
First layer (CONV1): 96 11x11 filters applied at stride 4 =>
Output volume [55x55x96]
Q: What if the input is now a small chunk of video? E.g. [227x227x3x15] ?
Extend the convolutional filters in time, perform spatio-temporal convolutions! E.g. can have 11x11xT filters, where T = 2..15.
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 12 29 Feb 2016
[3D Convolutional Neural Networks for Human Action Recognition, Ji et al., 2010]
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 13 29 Feb 2016
Sequential Deep Learning for Human Action Recognition, Baccouche et al., 2011
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 14 29 Feb 2016
spatio-temporal convolutions; worked best.
[Large-scale Video Classification with Convolutional Neural Networks, Karpathy et al., 2014]
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 15 29 Feb 2016
Learned filters on the first layer
[Large-scale Video Classification with Convolutional Neural Networks, Karpathy et al., 2014]
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 16 29 Feb 2016
1 million videos
487 sports classes
[Large-scale Video Classification with Convolutional Neural Networks, Karpathy et al., 2014]
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 17 29 Feb 2016
The motion information didn’t add all that much...
[Large-scale Video Classification with Convolutional Neural Networks, Karpathy et al., 2014]
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 18 29 Feb 2016
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 19 29 Feb 2016
3D VGGNet, basically.
[Learning Spatiotemporal Features with 3D Convolutional Networks, Tran et al. 2015]
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 20 29 Feb 2016
(of VGGNet fame)
[Two-Stream Convolutional Networks for Action Recognition in Videos, Simonyan and Zisserman 2014]
[T. Brox and J. Malik, “Large displacement optical flow: Descriptor matching in variational motion estimation,” 2011]
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 21 29 Feb 2016
Two-stream version works much better than either alone.
[Two-Stream Convolutional Networks for Action Recognition in Videos, Simonyan and Zisserman 2014]
[T. Brox and J. Malik, “Large displacement optical flow: Descriptor matching in variational motion estimation,” 2011]
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 22 29 Feb 2016
All 3D ConvNets so far used local motion cues to get extra accuracy (e.g. half a second or so)
what if the temporal dependencies of interest are much much longer? E.g. several seconds?
|
|
||||||
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 23 29 Feb 2016
(This paper was way ahead of its time. Cited 65 times.)
Sequential Deep Learning for Human Action Recognition, Baccouche et al., 2011
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 24 29 Feb 2016
LSTM way before it was cool
(This paper was way ahead of its time. Cited 65 times.)
Sequential Deep Learning for Human Action Recognition, Baccouche et al., 2011
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 25 29 Feb 2016
[Long-term Recurrent Convolutional Networks for Visual Recognition and Description, Donahue et al., 2015]
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 26 29 Feb 2016
[Beyond Short Snippets: Deep Networks for Video Classification, Ng et al., 2015]
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 27 29 Feb 2016
We looked at two types of architectural patterns:
Model temporal motion locally (3D CONV)
Model temporal motion globally (LSTM / RNN) + Fusions of both approaches at the same time.
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 28 29 Feb 2016
We looked at two types of architectural patterns:
Model temporal motion locally (3D CONV)
Model temporal motion globally (LSTM / RNN)
+ Fusions of both approaches at the same time.
There is another (cleaner) way!
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 29 29 Feb 2016
RNN
3D
CONVNET
Infinite (in theory)
temporal extent
(neurons that are function
of all video frames in the past)
Finite temporal
extent
(neurons that are only
a function of finitely many
video frames in the past)
video
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 30 29 Feb 2016
Beautiful:
All neurons in the ConvNet are recurrent.
Only requires (existing) 2D CONV routines. No need for 3D spatio-temporal CONV.
[Delving Deeper into Convolutional Networks for Learning Video Representations, Ballas et al., 2016]
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 31 29 Feb 2016
Normal ConvNet:
Convolution Layer
[Delving Deeper into Convolutional Networks for Learning Video Representations, Ballas et al., 2016]
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 32 29 Feb 2016
layer N
layer N+1 at previous timestep
CONV
RNN-like recurrence
(GRU)
CONV
layer N+1
[Delving Deeper into Convolutional Networks for Learning Video Representations, Ballas et al., 2016]
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 33 29 Feb 2016
Recall: RNNs Vanilla RNN
GRU LSTM
[Delving Deeper into Convolutional Networks for Learning Video Representations, Ballas et al., 2016]
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 34 29 Feb 2016
Recall: RNNs
GRU
Matrix multiply
=>
CONV
[Delving Deeper into Convolutional Networks for Learning Video Representations, Ballas et al., 2016]
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 35 29 Feb 2016
RNN
3D
CONVNET
Infinite (in theory)
temporal extent
(neurons that are function
of all video frames in the past)
Finite temporal
extent
(neurons that are only
a function of finitely many
video frames in the past)
video
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 36 29 Feb 2016
RNN
CONVNET
Infinite (in theory)
temporal extent
(neurons that are function
of all video frames in the past)
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 37 29 Feb 2016
You think you need a Spatio-Temporal Fancy Video ConvNet
STOP. Do you really?
Okay fine: do you want to model:
local motion? (use 3D CONV), or
global motion? (use LSTM).
Try out using Optical Flow in a second stream (can work better sometimes)
Try out GRU-RCN! (imo best model)
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 38 29 Feb 2016
Unsupervised Learning
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 39 29 Feb 2016
Definitions
Autoencoders
Vanilla
Variational
Adversarial Networks
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 40 29 Feb 2016
Supervised Learning
Data: (x, y)
x is data, y is label
Goal: Learn a function to
map x -> y
Examples: Classification,
regression, object detection,
semantic segmentation, image
captioning, etc
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 41 29 Feb 2016
Supervised Learning Data: (x, y)
x is data, y is label
Goal: Learn a function to map x -> y
Examples: Classification, regression, object detection, semantic segmentation, image captioning, etc
Unsupervised Learning Data: x
Just data, no labels!
Goal: Learn some structure of the data
Examples: Clustering, dimensionality reduction, feature learning, generative models, etc.
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 42 29 Feb 2016
Autoencoders
Traditional: feature learning
Variational: generate samples
Generative Adversarial Networks: Generate samples
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 43 29 Feb 2016
Features
Input data
z
Encoder
x
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 44 29 Feb 2016
Originally: Linear + nonlinearity (sigmoid)
Later: Deep, fully-connected
Later: ReLU CNN
Features
Input data
z
Encoder
x
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 45 29 Feb 2016
Autoencoders
z usually smaller than x (dimensionality reduction)
Originally: Linear + nonlinearity (sigmoid)
Later: Deep, fully-connected
Later: ReLU CNN
Features
Input data
z
Encoder
x
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 46 29 Feb 2016
| Reconstructed | ||
|
||
| input data | ||
Decoder
Features
Input data
z
Encoder
x
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 47 29 Feb 2016
| Autoencoders |
|
|
|
||
|
||
|
| Reconstructed | ||
|
||
| input data | ||
Decoder
Features z
Encoder
| Input data |
|
Encoder: 4-layer conv
Decoder: 4-layer upconv
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 48 29 Feb 2016
|
|
||||||||
|
|||||||||
|
|||||||||
|
|||||||||
| Reconstructed | |||||||||
|
|||||||||
| input data | |||||||||
|
|||||||||
| Encoder / decoder | |||||||||
| sometimes share | |||||||||
| weights |
|
|
|||||||
| Example: |
|
||||||||
| dim(x) = D | |||||||||
| dim(z) = H | |||||||||
| we: H x D | data |
|
|||||||
| w | d | : D x H = w T | |||||||
|
Train for
reconstruction with no labels!
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 49 29 Feb 2016
Loss function
(Often L2)
| Reconstructed |
|
|
| input data | ||
|
Features z
Encoder
| Input data |
|
Train for
reconstruction
with no labels!
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 50 29 Feb 2016
| Reconstructed | |||||||
| xx | |||||||
| input data | |||||||
| After training, |
|
||||||
| throw away | |||||||
| decoder! |
|
z | |||||
|
|||||||
| data | x |
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 51 29 Feb 2016
|
|
|||||
|
||||||
|
|
|
||||
| Use encoder to | ||||||
| initialize a |
|
|||||
| supervised | ||||||
| model | ||||||
|
||||||
|
Encoder
| Input data |
|
bird plane
dog deer truck
| Fine-tune |
|
| encoder |
|
| jointly with |
|
| classifier |
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 52 29 Feb 2016
In mid 2000s layer-wise pretraining with Restricted Boltzmann Machines (RBM) was common
Training deep nets was hard in 2006!
Hinton and Salakhutdinov, “Reducing the Dimensionality of Data with Neural Networks”, Science 2006
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 53 29 Feb 2016
In mid 2000s layer-wise pretraining with Restricted Boltzmann Machines (RBM) was common
Training deep nets was hard in 2006!
Hinton and Salakhutdinov, “Reducing the Dimensionality of Data with Neural Networks”, Science 2006
Not common anymore
With ReLU, proper initialization, batchnorm, Adam, etc easily train from scratch
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 54 29 Feb 2016
| Reconstructed | ||
|
||
| input data | ||
Decoder
Features z
Encoder
| Input data |
|
Autoencoders can reconstruct data, and can learn features to initialize a supervised model
Can we generate images from an autoencoder?
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 55 29 Feb 2016
A Bayesian spin on an autoencoder - lets us generate data!
Assume our data is generated like this:
| Sample from true | |||||
| conditional | |||||
| Sample from | |||||
|
|
||||
| true prior | |||||
Kingma and Welling, “Auto-Encoding
Variational Bayes”, ICLR 2014
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 56 29 Feb 2016
A Bayesian spin on an autoencoder!
Assume our data is generated like this:
Sample from true
conditional
| Sample from |
|
|
|
| true prior |
Intuition: x is an image, z gives class, orientation, attributes, etc
Kingma and Welling, “Auto-Encoding Variational Bayes”, ICLR 2014
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 57 29 Feb 2016
A Bayesian spin on an autoencoder!
Assume our data is generated like this:
Intuition: x is an image, z gives class, orientation, attributes, etc
| Sample from true |
|
||||||
| Sample from | conditional |
|
|||||
|
! | ||||||
|
|
||||||
| true prior | |||||||
Kingma and Welling, “Auto-Encoding
Variational Bayes”, ICLR 2014
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 58 29 Feb 2016
Prior: Assume
is a unit Gaussian
Kingma and Welling, ICLR 2014
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 59 29 Feb 2016
Prior: Assume
is a unit Gaussian
Conditional: Assume
is a
diagonal Gaussian,
predict mean and
variance with neural
net
Kingma and Welling, ICLR 2014
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 60 29 Feb 2016
Prior: Assume is a unit Gaussian
Conditional: Assume
is a
diagonal Gaussian,
predict mean and
variance with neural
net
Kingma and Welling, ICLR 2014
Mean and (diagonal) covariance of
x Σx
Decoder network
with parameters
z
Latent state
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 61 29 Feb 2016
Prior: Assume is a unit Gaussian
Conditional: Assume
is a
diagonal Gaussian,
predict mean and
variance with neural
net
Kingma and Welling, ICLR 2014
Mean and (diagonal) covariance of
x Σx
Decoder network
with parameters
z
Fully-connected or
Latent state
upconvolutional
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 62 29 Feb 2016
By Bayes Rule the posterior is:
Kingma and Welling,
ICLR 2014
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 63 29 Feb 2016
By Bayes Rule the posterior is:
Use decoder network =)
Gaussian =)
Intractible integral =(
Kingma and Welling,
ICLR 2014
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 64 29 Feb 2016
| By Bayes Rule the posterior is: |
|
|||||||||
|
||||||||||
| Use decoder network =) |
|
|
| Gaussian =) | ||
| Intractible integral =( |
|
Kingma and Welling,
ICLR 2014
x
Data point
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 65 29 Feb 2016
| By Bayes Rule the posterior is: |
|
|||||||||
|
||||||||||
| Use decoder network =) |
|
|
| Gaussian =) | ||
| Intractible integral =( |
|
Kingma and Welling, ICLR 2014
| Approximate posterior with | x | |||
| encoder network |
|
|||
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 66 29 Feb 2016
| By Bayes Rule the posterior is: |
|
|||||||||
|
||||||||||
|
||||||||||
|
||||||||||
| Use decoder network =) |
|
|
| Gaussian =) | ||
| Intractible integral =( |
|
Kingma and Welling, ICLR 2014
| Approximate posterior with | x | |||
| encoder network |
|
|||
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 67 29 Feb 2016
| Data point |
|
Kingma and Welling, ICLR 2014
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 68 29 Feb 2016
|
|||||||
|
|
||||||
|
|||||||
| Encoder network | |||||||
|
|
|
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 69 29 Feb 2016
| z | ||||||||
|
|
|||||||
|
|
|||||||
|
||||||||
| Encoder network | ||||||||
|
x |
|
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 70 29 Feb 2016
|
|
|
||||||
| Decoder network |
|
|||||||
| z | ||||||||
|
|
|||||||
|
|
|||||||
|
||||||||
|
||||||||
|
x |
|
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 71 29 Feb 2016
| Reconstructed | xx | |||||||||
|
||||||||||
|
|
|
||||||||
| Decoder network |
|
|||||||||
| z | ||||||||||
|
|
|||||||||
|
|
|||||||||
|
||||||||||
| Encoder network | ||||||||||
|
x |
|
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 72 29 Feb 2016
73 29 Feb 2016
Kingma and Welling, ICLR 2014
Reconstructed xx
Sample from
x Σx
Decoder network
| z | ||||||
|
||||||
|
|
|||||
| Encoder network | ||||||
|
x |
Training like a normal autoencoder: reconstruction loss at the end, regularization toward prior in middle
Mean and (diagonal) covariance of (should be close to data x)
Mean and (diagonal) covariance of (should be close
to prior )
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14
After network is trained:
z
Sample from
prior
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 74 29 Feb 2016
After network is trained:
x Σx
Decoder
network
z
Sample from
prior
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 75 29 Feb 2016
After network is trained:
Generated xx
Sample from
x Σx
Decoder
network
z
Sample from
prior
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 76 29 Feb 2016
After network is trained:
Generated xx
Sample from
x Σx
Decoder
network
z
Sample from
prior
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 77 29 Feb 2016
After network is trained:
Generated xx
Sample from
x Σx
Decoder
network
z
Sample from
prior
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 78 29 Feb 2016
After network is trained: Diagonal prior on z => independent latent variables
Generated xx
Sample from
x Σx
Decoder
network
z
Sample from
prior
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 79 29 Feb 2016
Variational Autoencoder: Math Maximum Likelihood?
Maximize likelihood of dataset
Kingma and Welling, ICLR 2014
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 80 29 Feb 2016
Variational Autoencoder: Math Maximum Likelihood?
Maximize likelihood of dataset
Maximize log-likelihood instead because sums are nicer
Kingma and Welling, ICLR 2014
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 81 29 Feb 2016
Variational Autoencoder: Math Maximum Likelihood?
Maximize likelihood of dataset
Maximize log-likelihood instead because sums are nicer
Marginalize joint
distribution
Kingma and Welling, ICLR 2014
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 82 29 Feb 2016
Variational Autoencoder: Math Maximum Likelihood?
Maximize likelihood of dataset
Maximize log-likelihood instead because sums are nicer
Intractible integral =(
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 83 29 Feb 2016
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 84 29 Feb 2016
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 85 29 Feb 2016
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 86 29 Feb 2016
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 87 29 Feb 2016
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 88 29 Feb 2016
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 89 29 Feb 2016
“Elbow”
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 90 29 Feb 2016
“Elbow”
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 91 29 Feb 2016
“Elbow”
Variational lower bound (elbow)
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 92 29 Feb 2016
Variational Autoencoder: Math
“Elbow”
|
|
|
|
|
|
Variational Autoencoder: Math
Reconstruct
the input
data
“Elbow”
|
|
|
|
|
|
| Variational Autoencoder: Math |
|
|
|
|
|
|
|
|
“Elbow”
|
|
|
|
|
|
Variational Autoencoder: Math
Reconstruct
the input
data
Sampling
with
reparam.
trick
(see paper)
“Elbow”
Latent states should follow the prior
|
|
|
|
|
|
Variational Autoencoder: Math
Reconstruct
the input
data
Sampling
with
reparam.
trick
(see paper)
“Elbow”
Latent states should follow the prior
Everything is Gaussian, closed form solution!
|
|
|
|
|
|
Traditional Autoencoders
Try to reconstruct input
Used to learn features, initialize supervised model
Not used much anymore
Variational Autoencoders
Bayesian meets deep learning
Sample from model to generate images
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 98 29 Feb 2016
Generative Adversarial Nets
Goodfellow et al, “Generative Adversarial Nets”, NIPS 2014
Can we generate images with less math?
| Random noise |
|
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 99 29 Feb 2016
Generative Adversarial Nets
Goodfellow et al, “Generative Adversarial Nets”, NIPS 2014
Can we generate images with less math?
| Fake image |
|
Generator
| Random noise |
|
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 100 29 Feb 2016
Generative Adversarial Nets
Goodfellow et al, “Generative Adversarial Nets”, NIPS 2014
Can we generate images with less math?
| Real or fake? |
|
Discriminator
| Fake image |
|
Generator
| Random noise |
|
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 101 29 Feb 2016
Generative Adversarial Nets
Goodfellow et al, “Generative Adversarial Nets”, NIPS 2014
Can we generate images with less math?
| Real or fake? |
|
Discriminator
| Fake image |
|
|||||||
|
|
|
||||||
|
||||||||
|
|
|||||||
|
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 102 29 Feb 2016
Generative Adversarial Nets
Goodfellow et al, “Generative Adversarial Nets”, NIPS 2014
| Real or fake? |
|
Discriminator
| Fake image |
|
Generator
| Random noise |
|
Can we generate images with less math?
Train generator and discriminator jointly After training, easy to generate images
|
|
Fake examples: from generator
Real examples: from dataset
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 103 29 Feb 2016
Generative Adversarial Nets
Generated samples
Nearest neighbor from training set
Goodfellow et al, “Generative Adversarial Nets”, NIPS 2014
Lecture 14 104 29 Feb 2016
Generative Adversarial Nets
Generated samples (CIFAR-10)
Nearest neighbor from training set
Goodfellow et al, “Generative Adversarial Nets”, NIPS 2014
Lecture 14 105 29 Feb 2016
Denton et al, “Deep generative image models using a Laplacian pyramid of adversarial networks”, NIPS 2015
Generate low-res
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 106 29 Feb 2016
Upsample
Denton et al, “Deep generative image models using a Laplacian pyramid of adversarial networks”, NIPS 2015
Generate low-res
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 107 29 Feb 2016
Upsample
|
|
|
|
||
|
||
| Denton et al, “Deep generative image models using a Laplacian pyramid of adversarial networks”, NIPS 2015 | ||
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 108 29 Feb 2016
Upsample Upsample
|
|
|
|
||
|
||
| Denton et al, “Deep generative image models using a Laplacian pyramid of adversarial networks”, NIPS 2015 | ||
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 109 29 Feb 2016
Upsample Upsample
|
|
|
|
|
|
||
|
|||
| Denton et al, “Deep generative image models using a Laplacian pyramid of adversarial networks”, NIPS 2015 | |||
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 110 29 Feb 2016
| Upsample |
|
|
|
|
|
|
|
|
|
||
|
|||
| Denton et al, “Deep generative image models using a Laplacian pyramid of adversarial networks”, NIPS 2015 | |||
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 111 29 Feb 2016
| Done! |
|
|
|
|
|
|
|
|
|
|
||||
|
|
|
||
| Denton et al, “Deep generative image models |
|
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 112 29 Feb 2016
Discriminators work
at every scale!
Denton et al, NIPS 2015
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 113 29 Feb 2016
Generative Adversarial Nets: Multiscale
Train separate model per-class on CIFAR-10
Denton et al, NIPS 2015
|
2016 |
Generator is an upsampling network with fractionally-strided convolutions Discriminator is a convolutional network
Radford et al, “Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks”, ICLR 2016
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 115 29 Feb 2016
Generator
Radford et al, “Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks”, ICLR 2016
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 116 29 Feb 2016
Samples from the model look amazing!
Radford et al, ICLR 2016
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 117 29 Feb 2016
Interpolating
between
random
points in latent
space
Radford et al, ICLR 2016
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 118 29 Feb 2016
Generative Adversarial Nets: Vector Math
Smiling woman Neutral woman Neutral man
Samples from the model
Radford et al, ICLR 2016
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 119 29 Feb 2016
Generative Adversarial Nets: Vector Math
Radford et al, ICLR 2016
Smiling woman Neutral woman Neutral man
Samples from the model
Average Z vectors, do arithmetic
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 120 29 Feb 2016
Generative Adversarial Nets: Vector Math
Smiling woman Neutral woman Neutral man
Radford et al, ICLR 2016
| Samples |
|
|
| from the | ||
| model |
Average Z vectors, do arithmetic
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 121 29 Feb 2016
Glasses man No glasses man No glasses woman
Radford et al, ICLR 2016
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 122 29 Feb 2016
Glasses man No glasses man No glasses woman
Woman with glasses
Radford et al, ICLR 2016
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 123 29 Feb 2016
Putting everything together
Pixel loss
xx
x Σx
Variational
Autoencoder z
z Σz
x
Dosovitskiy and Brox, “Generating Images with Perceptual Similarity Metrics based on Deep Networks”, arXiv 2016
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 124 29 Feb 2016
Putting everything together
Real or Generated
Dosovitskiy and Brox, “Generating Images with Perceptual Similarity Metrics based on Deep Networks”, arXiv 2016
| Discriminator |
|
|
| network |
|
|
xx
x Σx
Variational
Autoencoder z
z Σz
x
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 125 29 Feb 2016
Images with Perceptual Similarity
Putting everything together Metrics based on Deep Networks”, arXiv 2016
Real or Generated
Pretrained AlexNet
| Discriminator |
|
|
| network |
|
|
xx
x Σx
Variational
Autoencoder z
z Σz
x
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 126 29 Feb 2016
Images with Perceptual Similarity
Putting everything together Metrics based on Deep Networks”, arXiv 2016
Real or Generated
|
||||||
| Discriminator |
|
|||||
| network |
|
|||||
|
||||||
|
|
|
|
|||
|
|
|
||||
|
|
|||||
|
|
|
|
|||
x
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 127 29 Feb 2016
Dosovitskiy and Brox, “Generating
Images with Perceptual Similarity
Putting everything together Metrics based on Deep Networks”, arXiv 2016
Real or Generated
|
||||||
|
|
|||||
|
|
|||||
|
||||||
|
|
|
|
|||
|
|
|
||||
|
|
|||||
|
|
|
|
|||
|
|
|||||
| Fei-Fei Li & Andrej Karpathy & Justin Johnson |
|
|
Putting everything together
Samples from the model, trained on ImageNet
Dosovitskiy and Brox, “Generating Images with Perceptual Similarity Metrics based on Deep Networks”, arXiv 2016
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 129 29 Feb 2016
Videos
Unsupervised learning
Autoencoders: Traditional / variational
Generative Adversarial Networks
Next time: Guest lecture from Jeff Dean
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 130 29 Feb 2016


Java allows such qualification, but such accesses may indicate a subtle confusion of inheritance and overriding.
Example:
class Parent {
static int foo = 0;
}
class Child extends Parent { }
void bar() {
System.out.println(Child.foo);
}
After the quick-fix is applied, the result looks like this:
class Parent {
static int foo = 0;
}
class Child extends Parent { }
void bar() {
System.out.println(Parent.foo);
}