Tutorial¶
For deep learning, this tutorial will walk you through building a handwritten digits classifier using the MNIST dataset, arguably the “Hello World” of neural networks. For reinforcement learning, we will let computer learns to play Pong game from the original screen inputs. For nature language processing, we start from word embedding, and then describe language modeling and machine translation.
Note
For experts: Read the source code of InputLayer
and DenseLayer
, you
will understand how TensorLayer work. After that, we recommend you to read
the codes for tutorial directly.
Before we start¶
The tutorial assumes that you are somewhat familiar with neural networks and TensorFlow (the library which TensorLayer is built on top of). You can try to learn both at once from the Deeplearning Tutorial.
For a more slow-paced introduction to artificial neural networks, we recommend Convolutional Neural Networks for Visual Recognition by Andrej Karpathy et al., Neural Networks and Deep Learning by Michael Nielsen.
To learn more about TensorFlow, have a look at the TensorFlow tutorial. You will not need all of it, but a basic understanding of how TensorFlow works is required to be able to use TensorLayer. If you’re new to TensorFlow, going through that tutorial.
Run the MNIST example¶
In the first part of the tutorial, we will just run the MNIST example that’s included in the source distribution of TensorLayer. MNIST dataset contains 60000 handwritten digits that is commonly used for training various image processing systems, each of digit has 28x28 pixels.
We assume that you have already run through the Installation. If you
haven’t done so already, get a copy of the source tree of TensorLayer, and navigate
to the folder in a terminal window. Enter the folder and run the tutorial_mnist.py
example script:
python tutorial_mnist.py
If everything is set up correctly, you will get an output like the following:
tensorlayer: GPU MEM Fraction 0.300000
Downloading train-images-idx3-ubyte.gz
Downloading train-labels-idx1-ubyte.gz
Downloading t10k-images-idx3-ubyte.gz
Downloading t10k-labels-idx1-ubyte.gz
X_train.shape (50000, 784)
y_train.shape (50000,)
X_val.shape (10000, 784)
y_val.shape (10000,)
X_test.shape (10000, 784)
y_test.shape (10000,)
X float32 y int64
tensorlayer:Instantiate InputLayer input_layer (?, 784)
tensorlayer:Instantiate DropoutLayer drop1: keep: 0.800000
tensorlayer:Instantiate DenseLayer relu1: 800, <function relu at 0x11281cb70>
tensorlayer:Instantiate DropoutLayer drop2: keep: 0.500000
tensorlayer:Instantiate DenseLayer relu2: 800, <function relu at 0x11281cb70>
tensorlayer:Instantiate DropoutLayer drop3: keep: 0.500000
tensorlayer:Instantiate DenseLayer output_layer: 10, <function identity at 0x115e099d8>
param 0: (784, 800) (mean: -0.000053, median: -0.000043 std: 0.035558)
param 1: (800,) (mean: 0.000000, median: 0.000000 std: 0.000000)
param 2: (800, 800) (mean: 0.000008, median: 0.000041 std: 0.035371)
param 3: (800,) (mean: 0.000000, median: 0.000000 std: 0.000000)
param 4: (800, 10) (mean: 0.000469, median: 0.000432 std: 0.049895)
param 5: (10,) (mean: 0.000000, median: 0.000000 std: 0.000000)
num of params: 1276810
layer 0: Tensor("dropout/mul_1:0", shape=(?, 784), dtype=float32)
layer 1: Tensor("Relu:0", shape=(?, 800), dtype=float32)
layer 2: Tensor("dropout_1/mul_1:0", shape=(?, 800), dtype=float32)
layer 3: Tensor("Relu_1:0", shape=(?, 800), dtype=float32)
layer 4: Tensor("dropout_2/mul_1:0", shape=(?, 800), dtype=float32)
layer 5: Tensor("add_2:0", shape=(?, 10), dtype=float32)
learning_rate: 0.000100
batch_size: 128
Epoch 1 of 500 took 0.342539s
train loss: 0.330111
val loss: 0.298098
val acc: 0.910700
Epoch 10 of 500 took 0.356471s
train loss: 0.085225
val loss: 0.097082
val acc: 0.971700
Epoch 20 of 500 took 0.352137s
train loss: 0.040741
val loss: 0.070149
val acc: 0.978600
Epoch 30 of 500 took 0.350814s
train loss: 0.022995
val loss: 0.060471
val acc: 0.982800
Epoch 40 of 500 took 0.350996s
train loss: 0.013713
val loss: 0.055777
val acc: 0.983700
...
The example script allows you to try different models, including Multi-Layer Perceptron,
Dropout, Dropconnect, Stacked Denoising Autoencoder and Convolutional Neural Network.
Select different models from if __name__ == '__main__':
.
main_test_layers(model='relu')
main_test_denoise_AE(model='relu')
main_test_stacked_denoise_AE(model='relu')
main_test_cnn_layer()
Understand the MNIST example¶
Let’s now investigate what’s needed to make that happen! To follow along, open up the source code.
Preface¶
The first thing you might notice is that besides TensorLayer, we also import numpy and tensorflow:
import tensorflow as tf
import tensorlayer as tl
from tensorlayer.layers import set_keep
import numpy as np
import time
As we know, TensorLayer is built on top of TensorFlow, it is meant as a supplement helping
with some tasks, not as a replacement. You will always mix TensorLayer with some
vanilla TensorFlow code. The set_keep
is used to access the placeholder of keeping probabilities
when using Denoising Autoencoder.
Loading data¶
The first piece of code defines a function load_mnist_dataset()
. Its purpose is
to download the MNIST dataset (if it hasn’t been downloaded yet) and return it
in the form of regular numpy arrays. There is no TensorLayer involved at all, so
for the purpose of this tutorial, we can regard it as:
X_train, y_train, X_val, y_val, X_test, y_test = \
tl.files.load_mnist_dataset(shape=(-1,784))
X_train.shape
is (50000, 784)
, to be interpreted as: 50,000
images and each image has 784 pixels. y_train.shape
is simply (50000,)
, which is a vector the same
length of X_train
giving an integer class label for each image – namely,
the digit between 0 and 9 depicted in the image (according to the human
annotator who drew that digit).
For Convolutional Neural Network example, the MNIST can be load as 4D version as follow:
X_train, y_train, X_val, y_val, X_test, y_test = \
tl.files.load_mnist_dataset(shape=(-1, 28, 28, 1))
X_train.shape
is (50000, 28, 28, 1)
which represents 50,000 images with 1 channel, 28 rows and 28 columns each.
Channel one is because it is a grey scale image, every pixel have only one value.
Building the model¶
This is where TensorLayer steps in. It allows you to define an arbitrarily structured neural network by creating and stacking or merging layers. Since every layer knows its immediate incoming layers, the output layer (or output layers) of a network double as a handle to the network as a whole, so usually this is the only thing we will pass on to the rest of the code.
As mentioned above, tutorial_mnist.py
supports four types of models, and we
implement that via easily exchangeable functions of the same interface.
First, we’ll define a function that creates a Multi-Layer Perceptron (MLP) of
a fixed architecture, explaining all the steps in detail. We’ll then implement
a Denosing Autoencoder (DAE), after that we will then stack all Denoising Autoencoder and
supervised fine-tune them. Finally, we’ll show how to create a
Convolutional Neural Network (CNN).
Multi-Layer Perceptron (MLP)¶
The first script, main_test_layers()
, creates an MLP of two hidden layers of
800 units each, followed by a softmax output layer of 10 units. It applies 20%
dropout to the input data and 50% dropout to the hidden layers.
To feed data into the network, TensofFlow placeholders need to be defined as follow.
The None
here means the network will accept input data of arbitrary batchsize after compilation.
The x
is used to hold the X_train
data and y_
is used to hold the y_train
data.
If you know the batchsize beforehand and do not need this flexibility, you should give the batchsize
here – especially for convolutional layers, this can allow TensorFlow to apply
some optimizations.
x = tf.placeholder(tf.float32, shape=[None, 784], name='x')
y_ = tf.placeholder(tf.int64, shape=[None, ], name='y_')
The foundation of each neural network in TensorLayer is an
InputLayer
instance
representing the input data that will subsequently be fed to the network. Note
that the InputLayer
is not tied to any specific data yet.
network = tl.layers.InputLayer(x, name='input_layer')
Before adding the first hidden layer, we’ll apply 20% dropout to the input
data. This is realized via a DropoutLayer
instance:
network = tl.layers.DropoutLayer(network, keep=0.8, name='drop1')
Note that the first constructor argument is the incoming layer, the second
argument is the keeping probability for the activation value. Now we’ll proceed
with the first fully-connected hidden layer of 800 units. Note
that when stacking a DenseLayer
.
network = tl.layers.DenseLayer(network, n_units=800, act = tf.nn.relu, name='relu1')
Again, the first constructor argument means that we’re stacking network
on
top of network
.
n_units
simply gives the number of units for this fully-connected layer.
act
takes an activation function, several of which are defined
in tensorflow.nn
and tensorlayer.activation. Here we’ve chosen the rectifier, so
we’ll obtain ReLUs. We’ll now add dropout of 50%, another 800-unit dense layer and 50% dropout
again:
network = tl.layers.DropoutLayer(network, keep=0.5, name='drop2')
network = tl.layers.DenseLayer(network, n_units=800, act = tf.nn.relu, name='relu2')
network = tl.layers.DropoutLayer(network, keep=0.5, name='drop3')
Finally, we’ll add the fully-connected output layer which the n_units
equals to
the number of classes.
network = tl.layers.DenseLayer(network,
n_units=10,
act = tl.activation.identity,
name='output_layer')
As mentioned above, each layer is linked to its incoming layer(s), so we only need the output layer(s) to access a network in TensorLayer:
y = network.outputs
y_op = tf.argmax(tf.nn.softmax(y), 1)
cost = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(y, y_))
Here, network.outputs
is the 10 identity outputs from the network (in one hot format), y_op
is the integer
output represents the class index. While cost
is the cross-entropy between target and predicted labels.
Denoising Autoencoder (DAE)¶
Autoencoder is a unsupervised learning models which able to extract representative features, it has become more widely used for learning generative models of data and Greedy layer-wise pre-train. For vanilla Autoencoder see Deeplearning Tutorial.
The script main_test_denoise_AE()
implements a Denoising Autoencoder with corrosion rate of 50%.
The Autoencoder can be defined as follow, where an Autoencoder is represented by a DenseLayer
:
network = tl.layers.InputLayer(x, name='input_layer')
network = tl.layers.DropoutLayer(network, keep=0.5, name='denoising1')
network = tl.layers.DenseLayer(network, n_units=200, act=tf.nn.sigmoid, name='sigmoid1')
recon_layer1 = tl.layers.ReconLayer(network,
x_recon=x,
n_units=784,
act=tf.nn.sigmoid,
name='recon_layer1')
To train the DenseLayer
, simply run ReconLayer.pretrain()
, if using denoising Autoencoder, the name of
corrosion layer (a DropoutLayer
) need to be specified as follow. To save the feature images, set save
to True.
There are many kinds of pre-train metrices according to different architectures and applications. For sigmoid activation,
the Autoencoder can be implemented by using KL divergence, while for rectifer, L1 regularization of activation outputs
can make the output to be sparse. So the default behaviour of ReconLayer
only provide KLD and cross-entropy for sigmoid
activation function and L1 of activation outputs and mean-squared-error for rectifing activation function.
We recommend you to modify ReconLayer
to achieve your own pre-train metrice.
recon_layer1.pretrain(sess,
x=x,
X_train=X_train,
X_val=X_val,
denoise_name='denoising1',
n_epoch=200,
batch_size=128,
print_freq=10,
save=True,
save_name='w1pre_')
In addition, the script main_test_stacked_denoise_AE()
shows how to stacked multiple Autoencoder to one network and then
fine-tune.
Convolutional Neural Network (CNN)¶
Finally, the main_test_cnn_layer()
script creates two CNN layers and
max pooling stages, a fully-connected hidden layer and a fully-connected output
layer.
At the begin, we add a Conv2dLayer
with 32 filters of size 5x5 on top, follow by
max-pooling of factor 2 in both dimensions. And then apply a Conv2dLayer
with
64 filters of size 5x5 again and follow by a max_pool again. After that, flatten
the 4D output to 1D vector by using FlattenLayer
, and apply a dropout with 50%
to last hidden layer. The ?
represents arbitrary batch_size.
network = tl.layers.InputLayer(x, name='input_layer')
network = tl.layers.Conv2dLayer(network,
act = tf.nn.relu,
shape = [5, 5, 1, 32], # 32 features for each 5x5 patch
strides=[1, 1, 1, 1],
padding='SAME',
name ='cnn_layer1') # output: (?, 28, 28, 32)
network = tl.layers.PoolLayer(network,
ksize=[1, 2, 2, 1],
strides=[1, 2, 2, 1],
padding='SAME',
pool = tf.nn.max_pool,
name ='pool_layer1',) # output: (?, 14, 14, 32)
network = tl.layers.Conv2dLayer(network,
act = tf.nn.relu,
shape = [5, 5, 32, 64], # 64 features for each 5x5 patch
strides=[1, 1, 1, 1],
padding='SAME',
name ='cnn_layer2') # output: (?, 14, 14, 64)
network = tl.layers.PoolLayer(network,
ksize=[1, 2, 2, 1],
strides=[1, 2, 2, 1],
padding='SAME',
pool = tf.nn.max_pool,
name ='pool_layer2',) # output: (?, 7, 7, 64)
network = tl.layers.FlattenLayer(network, name='flatten_layer')
# output: (?, 3136)
network = tl.layers.DropoutLayer(network, keep=0.5, name='drop1')
# output: (?, 3136)
network = tl.layers.DenseLayer(network, n_units=256, act = tf.nn.relu, name='relu1')
# output: (?, 256)
network = tl.layers.DropoutLayer(network, keep=0.5, name='drop2')
# output: (?, 256)
network = tl.layers.DenseLayer(network, n_units=10, act = tl.identity, name='output_layer')
# output: (?, 10)
Note
For experts: Conv2dLayer
will create a convolutional layer using
tensorflow.nn.conv2d
, TensorFlow’s default convolution.
Training the model¶
The remaining part of the tutorial_mnist.py
script copes with setting up and running
a training loop over the MNIST dataset by using cross-entropy only.
Dataset iteration¶
An iteration function for synchronously iterating over two
numpy arrays of input data and targets, respectively, in mini-batches of a
given number of items. More iteration function can be found in tensorlayer.iterate
tl.iterate.minibatches(inputs, targets, batchsize, shuffle=False)
Loss and update expressions¶
Continuing, we create a loss expression to be minimized in training:
y = network.outputs
y_op = tf.argmax(tf.nn.softmax(y), 1)
cost = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(y, y_))
More cost or regularization can be applied here, take main_test_layers()
for example,
to apply max-norm on the weight matrices, we can add the following line:
cost = cost + tl.cost.maxnorm_regularizer(1.0)(network.all_params[0]) +
tl.cost.maxnorm_regularizer(1.0)(network.all_params[2])
Depending on the problem you are solving, you will need different loss functions,
see tensorlayer.cost
for more.
Having the model and the loss function defined, we create update expressions for training the network. TensorLayer do not provide many optimizer, we used TensorFlow’s optimizer instead:
train_params = network.all_params
train_op = tf.train.AdamOptimizer(learning_rate, beta1=0.9, beta2=0.999,
epsilon=1e-08, use_locking=False).minimize(cost, var_list=train_params)
For training the network, we fed data and the keeping probabilities to the feed_dict
.
feed_dict = {x: X_train_a, y_: y_train_a}
feed_dict.update( network.all_drop )
sess.run(train_op, feed_dict=feed_dict)
While, for validation and testing, we use slightly different way. All
dropout, dropconnect, corrosion layers need to be disable.
tl.utils.dict_to_one
set all network.all_drop
to 1.
dp_dict = tl.utils.dict_to_one( network.all_drop )
feed_dict = {x: X_test_a, y_: y_test_a}
feed_dict.update(dp_dict)
err, ac = sess.run([cost, acc], feed_dict=feed_dict)
As an additional monitoring quantity, we create an expression for the classification accuracy:
correct_prediction = tf.equal(tf.argmax(y, 1), y_)
acc = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
What Next?¶
We also have a more advanced image classification example in tutorial_cifar10.py
.
Please read the code and notes, figure out how to generate more training data and what
is local response normalization. After that, try to implement
Residual Network (Hint: you will need
to use the Layer.outputs).
Run the Pong Game example¶
In the second part of the tutorial, we will run the Deep Reinforcement Learning example that is introduced by Karpathy in Deep Reinforcement Learning: Pong from Pixels.
python tutorial_atari_pong.py
Before running the tutorial code, you need to install OpenAI gym environment which is a benchmark for Reinforcement Learning. If everything is set up correctly, you will get an output like the following:
[2016-07-12 09:31:59,760] Making new env: Pong-v0
tensorlayer:Instantiate InputLayer input_layer (?, 6400)
tensorlayer:Instantiate DenseLayer relu1: 200, <function relu at 0x1119471e0>
tensorlayer:Instantiate DenseLayer output_layer: 3, <function identity at 0x114bd39d8>
param 0: (6400, 200) (mean: -0.000009, median: -0.000018 std: 0.017393)
param 1: (200,) (mean: 0.000000, median: 0.000000 std: 0.000000)
param 2: (200, 3) (mean: 0.002239, median: 0.003122 std: 0.096611)
param 3: (3,) (mean: 0.000000, median: 0.000000 std: 0.000000)
num of params: 1280803
layer 0: Tensor("Relu:0", shape=(?, 200), dtype=float32)
layer 1: Tensor("add_1:0", shape=(?, 3), dtype=float32)
episode 0: game 0 took 0.17381s, reward: -1.000000
episode 0: game 1 took 0.12629s, reward: 1.000000 !!!!!!!!
episode 0: game 2 took 0.17082s, reward: -1.000000
episode 0: game 3 took 0.08944s, reward: -1.000000
episode 0: game 4 took 0.09446s, reward: -1.000000
episode 0: game 5 took 0.09440s, reward: -1.000000
episode 0: game 6 took 0.32798s, reward: -1.000000
episode 0: game 7 took 0.74437s, reward: -1.000000
episode 0: game 8 took 0.43013s, reward: -1.000000
episode 0: game 9 took 0.42496s, reward: -1.000000
episode 0: game 10 took 0.37128s, reward: -1.000000
episode 0: game 11 took 0.08979s, reward: -1.000000
episode 0: game 12 took 0.09138s, reward: -1.000000
episode 0: game 13 took 0.09142s, reward: -1.000000
episode 0: game 14 took 0.09639s, reward: -1.000000
episode 0: game 15 took 0.09852s, reward: -1.000000
episode 0: game 16 took 0.09984s, reward: -1.000000
episode 0: game 17 took 0.09575s, reward: -1.000000
episode 0: game 18 took 0.09416s, reward: -1.000000
episode 0: game 19 took 0.08674s, reward: -1.000000
episode 0: game 20 took 0.09628s, reward: -1.000000
resetting env. episode reward total was -20.000000. running mean: -20.000000
episode 1: game 0 took 0.09910s, reward: -1.000000
episode 1: game 1 took 0.17056s, reward: -1.000000
episode 1: game 2 took 0.09306s, reward: -1.000000
episode 1: game 3 took 0.09556s, reward: -1.000000
episode 1: game 4 took 0.12520s, reward: 1.000000 !!!!!!!!
episode 1: game 5 took 0.17348s, reward: -1.000000
episode 1: game 6 took 0.09415s, reward: -1.000000
This example allow computer to learn how to play Pong game from the screen inputs, just like human behavior. After training for 15,000 episodes, the computer can win 20% of the games. The computer win 35% of the games at 20,000 episode, we can seen the computer learn faster and faster as it has more winning data to train. If you run it for 30,000 episode, it start to win.
render = False
resume = False
Setting ‘render’ to ‘True’, if you want to display the game environment. When you run the code again, you can set ‘resume’ to ‘True’, the code will load the existing model and train the model basic on it.
Understand Reinforcement learning¶
Pong Game¶
To understand Reinforcement Learning, we let computer to learn how to play Pong game from the original screen inputs. Before we start, we highly recommend you to go through a famous blog called Deep Reinforcement Learning: Pong from Pixels which is a minimalistic implementation of Deep Reinforcement Learning by using python-numpy and OpenAI gym environment.
python tutorial_atari_pong.py
Policy Network¶
In Deep Reinforcement Learning, the Policy Network is the same with Deep Neural Network, it is our player (or “agent”) who output actions to tell what we should do (move UP or DOWN); in Karpathy’s code, he only defined 2 actions, UP and DOWN and using a single simgoid output; In order to make our tutorial more generic, we defined 3 actions which are UP, DOWN and STOP (do nothing) by using 3 softmax outputs.
# observation for training
states_batch_pl = tf.placeholder(tf.float32, shape=[None, D])
network = tl.layers.InputLayer(states_batch_pl, name='input_layer')
network = tl.layers.DenseLayer(network, n_units=H,
act = tf.nn.relu, name='relu1')
network = tl.layers.DenseLayer(network, n_units=3,
act = tl.activation.identity, name='output_layer')
probs = network.outputs
sampling_prob = tf.nn.softmax(probs)
Then when our agent is playing Pong, it calculates the probabilities of different actions, and then draw sample (action) from this uniform distribution. As the actions are represented by 1, 2 and 3, but the softmax outputs should be start from 0, we calculate the label value by minus 1.
prob = sess.run(
sampling_prob,
feed_dict={states_batch_pl: x}
)
# action. 1: STOP 2: UP 3: DOWN
action = np.random.choice([1,2,3], p=prob.flatten())
...
ys.append(action - 1)
Policy Gradient¶
The key of Deep Reinforcement Learning is how to train the Policy Network, there are many way to do
Q-learning xxxxx
AlphaGo is using xxxx
Dataset iteration¶
In Reinforcement Learning, we consider a final decision as an episode. In Pong game, a episode is a few dozen games, because the games go up to score of 21 for either player. Then the batch size is how many episode we consider to update the model. In the tutorial, we train a 2-layer policy network with 200 hidden layer units using RMSProp on batches of 10 episodes.
Loss and update expressions¶
Continuing, we create a loss expression to be minimized in training:
actions_batch_pl = tf.placeholder(tf.int32, shape=[None])
discount_rewards_batch_pl = tf.placeholder(tf.float32, shape=[None])
loss = tl.rein.cross_entropy_reward_loss(probs, actions_batch_pl,
discount_rewards_batch_pl)
...
...
sess.run(
train_op,
feed_dict={
states_batch_pl: epx,
actions_batch_pl: epy,
discount_rewards_batch_pl: disR
}
)
The loss in a batch is relate to all outputs of Policy Network, all actions we made and the corresponding discounted rewards in a batch. We first compute the loss of each action by multiplying the discounted reward and the cross-entropy between its output and its true action. The final loss in a batch is the sum of all loss of the actions.
What Next?¶
The tutorial above shows how you can build your own agent, end-to-end. While it has reasonable quality, the default parameters will not give you the best agent model. Here are a few things you can improve.
First of all, instead of conventional MLP model, we can use CNNs to capture the screen information better as Playing Atari with Deep Reinforcement Learning describe.
Also, the default parameters of the model are not tuned. You can try changing the learning rate, decay, or initializing the weights of your model in a different way.
Finally, you can try the model on different tasks (games).
Run the Word2Vec example¶
In this part of the tutorial, we train a matrix for words, where each word can be represented by a unique row vector in the matrix. In the end, similar words will have similar vectors. Then as we plot out the words into a two-dimensional plane, words that are similar end up clustering nearby each other
python tutorial_word2vec_basic.py
If everything is set up correctly, you will get an output in the end.
Understand Word Embedding¶
Word Embedding¶
Hao Dong highly recommend you to read Colah’s blog Word Representations to understand why we want to use a vector representation, and how to compute the vectors.
Train an embedding matrix
# train_inputs is a row vector, a input is an integer id of single word.
# train_labels is a column vector, a label is an integer id of single word.
# valid_dataset is a column vector, a valid set is an integer id of single word.
train_inputs = tf.placeholder(tf.int32, shape=[batch_size])
train_labels = tf.placeholder(tf.int32, shape=[batch_size, 1])
valid_dataset = tf.constant(valid_examples, dtype=tf.int32)
# Look up embeddings for inputs.
emb_net = tl.layers.Word2vecEmbeddingInputlayer(
inputs = train_inputs,
train_labels = train_labels,
vocabulary_size = vocabulary_size,
embedding_size = embedding_size,
num_sampled = num_sampled,
nce_loss_args = {},
E_init = tf.random_uniform_initializer(minval=-1.0, maxval=1.0),
E_init_args = {},
nce_W_init = tf.truncated_normal_initializer(
stddev=float(1.0/np.sqrt(embedding_size))),
nce_W_init_args = {},
nce_b_init = tf.constant_initializer(value=0.0),
nce_b_init_args = {},
name ='word2vec_layer',
)
cost = emb_net.nce_cost
Dataset iteration¶
Loss and update expressions¶
Load an Embedding matrix¶
In the end of training the embedding matrix,
python tutorial_generate_text.py
load_params = tl.files.load_npz(name=model_file_name+'.npz')
x = tf.placeholder(tf.int32, shape=[batch_size])
y_ = tf.placeholder(tf.int32, shape=[batch_size, 1])
emb_net = tl.layers.EmbeddingInputlayer(
inputs = x,
vocabulary_size = vocabulary_size,
embedding_size = embedding_size,
name ='embedding_layer')
sess.run(tf.initialize_all_variables())
tl.files.assign_params(sess, [load_params[0]], emb_net)
Run the PTB example¶
Penn TreeBank (PTB) dataset is used in many LANGUAGE MODELING papers, including “Empirical Evaluation and Combination of Advanced Language Modeling Techniques”, “Recurrent Neural Network Regularization”. It consists of 929k training words, 73k validation words, and 82k test words. It has 10k words in its vocabulary.
The PTB example is trying to show how to train a recurrent neural network on a challenging task of language modeling.
Given a sentence “I am from Imperial College London”, the model can learn to
predict “Imperial College London” from “from Imperial College”. In other
word, it predict next words in a text given a history of previous words.
In this case, num_steps (sequence length)
is 3.
python tutorial_ptb_lstm.py
The script provides three settings (small, medium, large), larger model has better performance, you can choice different setting in:
flags.DEFINE_string(
"model", "small",
"A type of model. Possible options are: small, medium, large.")

Epoch: 13 Learning rate: 0.004
0.004 perplexity: 56.122 speed: 8594 wps
0.104 perplexity: 40.793 speed: 9186 wps
0.204 perplexity: 44.527 speed: 9117 wps
0.304 perplexity: 42.668 speed: 9214 wps
0.404 perplexity: 41.943 speed: 9269 wps
0.504 perplexity: 41.286 speed: 9271 wps
0.604 perplexity: 39.989 speed: 9244 wps
0.703 perplexity: 39.403 speed: 9236 wps
0.803 perplexity: 38.742 speed: 9229 wps
0.903 perplexity: 37.430 speed: 9240 wps
Epoch: 13 Train Perplexity: 36.643
Epoch: 13 Valid Perplexity: 121.475
Test Perplexity: 116.716
The PTB example proves RNN is able to modeling language, but this example
did not do something practical. However, you should read through this example
and Understand LSTM
in order to understand the basic of RNN.
After that, you learn how to generate text, how to achieve language translation
and how to build a questions answering system by using RNN.
Understand LSTM¶
Recurrent Neural Network¶
Hao Dong personally think Andrey Karpathy’s blog is the best material to Understand Recurrent Neural Network , after reading that, Colah’s blog can help you to Understand LSTM Network which can solve The Problem of Long-Term Dependencies. We do not describe more about RNN, please read through these blogs before you go on.
Image by Andrey Karpathy
Synced sequence input and output¶
The model in PTB example is a typically type of synced sequence input and output, which was described by Karpathy as “(5) Synced sequence input and output (e.g. video classification where we wish to label each frame of the video). Notice that in every case are no pre-specified constraints on the lengths sequences because the recurrent transformation (green) is fixed and can be applied as many times as we like.”
The model is built as follow. Firstly, transfer the words into word vectors by looking up an embedding matrix. In this tutorial, no pre-training on embedding matrix. Secondly, we stacked two LSTMs together use dropout among the embedding layer, LSTM layers and output layer for regularization. The model provides a sequence of softmax outputs during training.
The first LSTM layer outputs [batch_size, num_steps, hidden_size] for stacking another LSTM after it. The second LSTM layer outputs [batch_size*num_steps, hidden_size] for stacking DenseLayer after it, then compute the softmax outputs of each example, i.e. n_examples = batch_size*num_steps.
To understand the PTB tutorial, you can also read TensorFlow PTB tutorial.
network = tl.layers.EmbeddingInputlayer(
inputs = x,
vocabulary_size = vocab_size,
embedding_size = hidden_size,
E_init = tf.random_uniform_initializer(-init_scale, init_scale),
name ='embedding_layer')
if is_training:
network = tl.layers.DropoutLayer(network, keep=keep_prob, name='drop1')
network = tl.layers.RNNLayer(network,
cell_fn=tf.nn.rnn_cell.BasicLSTMCell,
cell_init_args={'forget_bias': 0.0},
n_hidden=hidden_size,
initializer=tf.random_uniform_initializer(-init_scale, init_scale),
n_steps=num_steps,
return_last=False,
name='basic_lstm_layer1')
lstm1 = network
if is_training:
network = tl.layers.DropoutLayer(network, keep=keep_prob, name='drop2')
network = tl.layers.RNNLayer(network,
cell_fn=tf.nn.rnn_cell.BasicLSTMCell,
cell_init_args={'forget_bias': 0.0},
n_hidden=hidden_size,
initializer=tf.random_uniform_initializer(-init_scale, init_scale),
n_steps=num_steps,
return_last=False,
return_seq_2d=True,
name='basic_lstm_layer2')
lstm2 = network
if is_training:
network = tl.layers.DropoutLayer(network, keep=keep_prob, name='drop3')
network = tl.layers.DenseLayer(network,
n_units=vocab_size,
W_init=tf.random_uniform_initializer(-init_scale, init_scale),
b_init=tf.random_uniform_initializer(-init_scale, init_scale),
act = tl.activation.identity, name='output_layer')
Dataset iteration¶
The batch_size can be seem as how many concurrent computations. As the following example shows, the first batch learn the sequence information by using 0 to 9. The second batch learn the sequence information by using 10 to 19. So it ignores the information from 9 to 10 !n If only if we set the batch_size = 1, it will consider all information from 0 to 20.
The meaning of batch_size here is not the same with the MNIST example. In MNIST example, batch_size reflects how many examples we consider in each iteration, while in PTB example, batch_size is how many concurrent processes (segments) for speed up computation.
Some Information will be ignored if batch_size > 1, however, if your dataset is “long” enough (a text corpus usually has billions words), the ignored information would not effect the final result.
In PTB tutorial, we setted batch_size = 20, so we cut the dataset into 20 segments. At the begining of each epoch, we initialize (reset) the 20 RNN states for 20 segments, then go through 20 segments separately.
The training data will be generated as follow:
train_data = [i for i in range(20)]
for batch in tl.iterate.ptb_iterator(train_data, batch_size=2, num_steps=3):
x, y = batch
print(x, '\n',y)
... [[ 0 1 2] <---x 1st subset/ iteration
... [10 11 12]]
... [[ 1 2 3] <---y
... [11 12 13]]
...
... [[ 3 4 5] <--- 1st batch input 2nd subset/ iteration
... [13 14 15]] <--- 2nd batch input
... [[ 4 5 6] <--- 1st batch target
... [14 15 16]] <--- 2nd batch target
...
... [[ 6 7 8] 3rd subset/ iteration
... [16 17 18]]
... [[ 7 8 9]
... [17 18 19]]
Note
This example can also be considered as pre-training of the word embedding matrix.
Loss and update expressions¶
The cost function is the averaged cost of each mini-batch:
def loss_fn(outputs, targets, batch_size, num_steps):
# Returns the cost function of Cross-entropy of two sequences, implement
# softmax internally.
# outputs : 2D tensor [batch_size*num_steps, n_units of output layer]
# targets : 2D tensor [batch_size, num_steps], need to be reshaped.
# n_examples = batch_size * num_steps
# so
# cost is the averaged cost of each mini-batch (concurrent process).
loss = tf.nn.seq2seq.sequence_loss_by_example(
[outputs],
[tf.reshape(targets, [-1])],
[tf.ones([batch_size * num_steps])])
cost = tf.reduce_sum(loss) / batch_size
return cost
# Cost for Training
cost = loss_fn(network.outputs, targets, batch_size, num_steps)
For updating, this example decreases the initial learning rate after several
epoachs (defined by max_epoch
), by multipling a lr_decay
. In addition,
truncated backpropagation clips values of gradients by the ratio of the sum of
their norms, so as to make the learning process tractable.
# Truncated Backpropagation for training
with tf.variable_scope('learning_rate'):
lr = tf.Variable(0.0, trainable=False)
tvars = tf.trainable_variables()
grads, _ = tf.clip_by_global_norm(tf.gradients(cost, tvars),
max_grad_norm)
optimizer = tf.train.GradientDescentOptimizer(lr)
train_op = optimizer.apply_gradients(zip(grads, tvars))
Then at the beginning of each epoch, we assign a new learning rate:
new_lr_decay = lr_decay ** max(i - max_epoch, 0.0)
sess.run(tf.assign(lr, learning_rate * new_lr_decay))
At the begining of each epoch, all states of LSTMs need to be resetted (initialized), then after each iteration, the new final states need to be assigned as the initial states of next iteration:
state1 = tl.layers.initialize_rnn_state(lstm1.initial_state)
state2 = tl.layers.initialize_rnn_state(lstm2.initial_state)
for step, (x, y) in enumerate(tl.iterate.ptb_iterator(train_data,
batch_size, num_steps)):
feed_dict = {input_data: x, targets: y,
lstm1.initial_state: state1,
lstm2.initial_state: state2,
}
# For training, enable dropout
feed_dict.update( network.all_drop )
_cost, state1, state2, _ = sess.run([cost,
lstm1.final_state,
lstm2.final_state,
train_op],
feed_dict=feed_dict
)
costs += _cost; iters += num_steps
Predicting¶
After training the model, we no long consider the number of steps (sequence length),
i.e. batch_size, num_steps
are 1
. Then we can predict the next word step
by step, instead of predict a sequence of words from a sequence of words.
input_data_test = tf.placeholder(tf.int32, [1, 1])
targets_test = tf.placeholder(tf.int32, [1, 1])
...
network_test, lstm1_test, lstm2_test = inference(input_data_test,
is_training=False, num_steps=1, reuse=True)
...
cost_test = loss_fn(network_test.outputs, targets_test, 1, 1)
...
print("Evaluation")
# Testing
# go through the test set step by step, it will take a while.
start_time = time.time()
costs = 0.0; iters = 0
# reset all states at the begining
state1 = tl.layers.initialize_rnn_state(lstm1_test.initial_state)
state2 = tl.layers.initialize_rnn_state(lstm2_test.initial_state)
for step, (x, y) in enumerate(tl.iterate.ptb_iterator(test_data,
batch_size=1, num_steps=1)):
feed_dict = {input_data_test: x, targets_test: y,
lstm1_test.initial_state: state1,
lstm2_test.initial_state: state2,
}
_cost, state1, state2 = sess.run([cost_test,
lstm1_test.final_state,
lstm2_test.final_state],
feed_dict=feed_dict
)
costs += _cost; iters += 1
test_perplexity = np.exp(costs / iters)
print("Test Perplexity: %.3f took %.2fs" % (test_perplexity, time.time() - start_time))
What Next?¶
Now, you understand Synced sequence input and output. Let think about Many to one (Sequence input and one output), we can also use “I am from Imperial” to predict the next word “College” right? Please try your best to build a text generator, which give some seed words to generate context, some people even used Many to one model to automatically generate papers !
Karpathy’s blog : “(3) Sequence input (e.g. sentiment analysis where a given sentence is classified as expressing positive or negative sentiment). “
Run the Translation example¶
python tutorial_translate.py
This script is going to training a neural network to translate English to French. If everything is correct, you will see.
- Download WMT English-to-French translation data, includes training and testing data.
- Create vocabulary files for English and French from training data.
- Create the tokenized training and testing data from original training and testing data.
Load or Download WMT English-to-French translation > wmt
wmt/giga-fren.release2
wmt/newstest2013
wmt/vocab40000.fr
wmt/vocab40000.en
Creating vocabulary wmt/vocab40000.fr from data wmt/giga-fren.release2.fr
processing line 100000
processing line 200000
processing line 300000
processing line 400000
processing line 500000
processing line 600000
processing line 700000
processing line 800000
processing line 900000
processing line 1000000
processing line 1100000
processing line 1200000
...
processing line 22500000
Creating vocabulary wmt/vocab40000.en from data wmt/giga-fren.release2.en
processing line 100000
...
processing line 22500000
...
Firstly, we download English-to-French translation data from the WMT‘15 Website. The training and testing data as follow. The training data is used to train the model, the testing data is used to XXXX.
wmt/training-giga-fren.tar <-- Training data for English-to-French (2.6GB)
giga-fren.release2.* are extracted from it.
wmt/dev-v2.tgz <-- Testing data for different language (21.4MB)
newstest2013.* are extracted from it.
wmt/giga-fren.release2.fr <-- Training data of French (4.57GB)
wmt/giga-fren.release2.en <-- Training data of English (3.79GB)
wmt/newstest2013.fr <-- Testing data of French (393KB)
wmt/newstest2013.en <-- Testing data of English (333KB)
As giga-fren.release2.*
are the training data, giga-fren.release2.fr
look as follow.
Il a transformé notre vie | Il a transformé la société | Son fonctionnement | La technologie, moteur du changement Accueil | Concepts | Enseignants | Recherche | Aperçu | Collaborateurs | Web HHCC | Ressources | Commentaires Musée virtuel du Canada
Plan du site
Rétroaction
Crédits
English
Qu’est-ce que la lumière?
La découverte du spectre de la lumière blanche Des codes dans la lumière Le spectre électromagnétique Les spectres d’émission Les spectres d’absorption Les années-lumière La pollution lumineuse
Le ciel des premiers habitants La vision contemporaine de l'Univers L’astronomie pour tous
Bande dessinée
Liens
Glossaire
Observatoires
...
While giga-fren.release2.en
look as follow, we can see words or sentences
are separated by |
or \n
.
Changing Lives | Changing Society | How It Works | Technology Drives Change Home | Concepts | Teachers | Search | Overview | Credits | HHCC Web | Reference | Feedback Virtual Museum of Canada Home Page
Site map
Feedback
Credits
Français
What is light ?
The white light spectrum Codes in the light The electromagnetic spectrum Emission spectra Absorption spectra Light-years Light pollution
The sky of the first inhabitants A contemporary vison of the Universe Astronomy for everyone
Cartoon
Links
Glossary
Observatories
The testing data newstest2013.en
and newstest2013.fr
look as follow.
newstest2013.en :
A Republican strategy to counter the re-election of Obama
Republican leaders justified their policy by the need to combat electoral fraud.
However, the Brennan Centre considers this a myth, stating that electoral fraud is rarer in the United States than the number of people killed by lightning.
newstest2013.fr :
Une stratégie républicaine pour contrer la réélection d'Obama
Les dirigeants républicains justifièrent leur politique par la nécessité de lutter contre la fraude électorale.
Or, le Centre Brennan considère cette dernière comme un mythe, affirmant que la fraude électorale est plus rare aux États-Unis que le nombre de personnes tuées par la foudre.
After downloading the dataset, it start to create vocabulary files,
vocab40000.fr
and vocab40000.en
from the training data giga-fren.release2.fr
and giga-fren.release2.en
, usually it will take a while. The number 40000
reflects the vocabulary size.
The vocab40000.fr
(381KB) stores one-item-per-line as follow.
_PAD
_GO
_EOS
_UNK
de
,
.
'
la
et
des
les
à
le
du
l
en
)
d
0
(
00
pour
dans
un
que
une
sur
au
0000
a
par
The vocab40000.en
(344KB) stores one-item-per-line as follow.
_PAD
_GO
_EOS
_UNK
the
.
,
of
and
to
in
a
)
(
0
for
00
that
is
on
The
0000
be
by
with
or
:
as
"
000
are
;
And then, we start to create the tokenized training and testing data for both English and French. It will take a while as well.
Tokenizing data in wmt/giga-fren.release2.fr <-- Training data of French
tokenizing line 100000
tokenizing line 200000
tokenizing line 300000
tokenizing line 400000
...
tokenizing line 22500000
Tokenizing data in wmt/giga-fren.release2.en <-- Training data of English
tokenizing line 100000
tokenizing line 200000
tokenizing line 300000
tokenizing line 400000
...
tokenizing line 22500000
Tokenizing data in wmt/newstest2013.fr <-- Testing data of French
Tokenizing data in wmt/newstest2013.en <-- Testing data of English
In the end, all files we have as follow.
wmt/training-giga-fren.tar <-- Compressed Training data for English-to-French (2.6GB)
giga-fren.release2.* are extracted from it.
wmt/dev-v2.tgz <-- Compressed Testing data for different language (21.4MB)
newstest2013.* are extracted from it.
wmt/giga-fren.release2.fr <-- Training data of French (4.57GB)
wmt/giga-fren.release2.en <-- Training data of English (3.79GB)
wmt/newstest2013.fr <-- Testing data of French (393KB)
wmt/newstest2013.en <-- Testing data of English (333KB)
wmt/vocab40000.fr <-- Vocabulary of French (381KB)
wmt/vocab40000.en <-- Vocabulary of English (344KB)
wmt/giga-fren.release2.ids40000.fr <-- Tokenized Training data of French (2.81GB)
wmt/giga-fren.release2.ids40000.en <-- Tokenized Training data of English (2.38GB)
wmt/newstest2013.ids40000.fr <-- Tokenized Testing data of French (268KB)
wmt/newstest2013.ids40000.en <-- Tokenized Testing data of English (232KB)
Now, read all tokenized data into buckets and compute their sizes.
Reading development (testing) data into buckets
dev data: (5, 10) [[13388, 4, 949], [23113, 8, 910, 2]]
en word_ids: [13388, 4, 949]
en context: [b'Preventing', b'the', b'disease']
fr word_ids: [23113, 8, 910, 2]
fr context: [b'Pr\xc3\xa9venir', b'la', b'maladie', b'_EOS']
Reading training data into buckets (limit: 0).
reading data line 100000
reading data line 200000
reading data line 300000
reading data line 400000
reading data line 500000
reading data line 600000
reading data line 700000
reading data line 800000
...
reading data line 22400000
reading data line 22500000
train_bucket_sizes: [239121, 1344322, 5239557, 10445326]
train_total_size: 17268326.0
train_buckets_scale: [0.013847375825543252, 0.09169638099257565, 0.3951164693091849, 1.0]
train data: (5, 10) [[1368, 3344], [1089, 14, 261, 2]]
en word_ids: [1368, 3344]
en context: [b'Site', b'map']
fr word_ids: [1089, 14, 261, 2]
fr context: [b'Plan', b'du', b'site', b'_EOS']
Start training by using the tokenized bucket data, the training process can
only be terminated by stop the program.

After training the model, you can play with the
Understand Translation¶
Seq2seq¶
Basics¶
Sequence to sequence is a type of “Many to many” but different with Synced sequence input and output in PTB tutorial. Seq2seq generates sequence output after feeding all sequence inputs. The following two methods can improve the accuracy:
- Reversing the inputs
- Attention mechanism
To speed up the computation, we used:
- Sampled softmax
Karpathy’s blog described Seq2seq as: “(4) Sequence input and sequence output (e.g. Machine Translation: an RNN reads a sentence in English and then outputs a sentence in French).”
As the above figure shows, the encoder inputs, decoder inputs and targets are:
encoder_input = A B C
decoder_input = <go> W X Y Z
targets = W X Y Z <eos>
Note: in the code, the size of targets is one smaller than the size
of decoder_input, not like this figure.
Papers¶
The English-to-French example implements a multi-layer recurrent neural network as encoder, and an Attention-based decoder. It is the same as the model described in this paper:
The example uses sampled softmax to handle large output vocabulary size.
In this case, as target_vocab_size=4000
, for vocabularies smaller
than 512
, it might be a better idea to just use a standard softmax loss.
Sampled softmax is described in Section 3 of the this paper:
Reversing the inputs and Multi-layer cells have been successfully used in sequence-to-sequence models for translation has beed described in this paper:
Attention mechanism allows the decoder more direct access to the input, it was described in this paper:
Alternatively, the model can also be implemented by a single-layer version, but with Bi-directional encoder, was presented in this paper:
Implementation¶
Bucketing and Padding¶
Bucketing is a method to efficiently handle sentences of different length.
When translating English to French, we will have English sentences of
different lengths L1
on input, and French sentences of different
lengths L2
on output. We should in principle create a seq2seq model
for every pair (L1, L2+1)
(prefixed by a GO symbol) of
lengths of an English and French sentence.
For find the closest bucket for each pair, then we could just pad every sentence with a special PAD symbol in the end if the bucket is bigger than the sentence
We use a number of buckets and pad to the closest one for efficiency. In this example, we used 4 buckets.
buckets = [(5, 10), (10, 15), (20, 25), (40, 50)]
If the input is an English sentence with 3
tokens, and the corresponding
output is a French sentence with 6
tokens, then they will be put in the
first bucket and padded to length 5
for encoder inputs (English sentence),
and length 10
for decoder inputs.
If we have an English sentence with 8 tokens and the corresponding French
sentence has 18 tokens, then they will be fit into (20, 25)
bucket.
In other word, bucket (I, O)
is (encoder_input_size, decoder_inputs_size)
.
Given a pair of [["I", "go", "."], ["Je", "vais", "."]]
in tokenized format,
we fit it into bucket (5, 10)
.
The training data of encoder inputs representing [PAD PAD "." "go" "I"]
and decoder inputs [GO "Je" "vais" "." EOS PAD PAD PAD PAD PAD]
. The targets
are decoder inputs shifted by one.
bucket = (I, O) = (5, 10)
encoder_inputs = [PAD PAD "." "go" "I"] <-- I
decoder_inputs = [GO "Je" "vais" "." EOS PAD PAD PAD PAD PAD] <-- O
targets = ["Je" "vais" "." EOS PAD PAD PAD PAD PAD] <-- O - 1
target_weights = ???? <-- O
Special vocabulary symbols, punctuation and digits¶
The special vocabulary symbols in this example are:
_PAD = b"_PAD"
_GO = b"_GO"
_EOS = b"_EOS"
_UNK = b"_UNK"
PAD_ID = 0 <-- index (row number) in vocabulary
GO_ID = 1
EOS_ID = 2
UNK_ID = 3
_START_VOCAB = [_PAD, _GO, _EOS, _UNK]
ID MEANINGS
_PAD 0 Padding, empty word
_GO 1 1st element of decoder_inputs
_EOS 2 End of Sentence of targets
_UNK 3 Unknown word, words do not exist in vocabulary will be marked as 3
For digits, the normalize_digits
of creating vocabularies and tokenized dataset
should be consistent, if True
all digits will be replaced by 0
. Like
123
to 000`
, 9 to 0 and 19762 to 00000, then 0, 00, 000
will be the words in the vocabulary. Otherwise, if False
, different digits
will be seem in the vocabulary, then the vocabulary size will be very big.
The regular expression to find digits is _DIGIT_RE = re.compile(br"\d")
,
see tl.files.create_vocabulary()
and tl.files.data_to_token_ids()
.
For word split, the regular expression is
_WORD_SPLIT = re.compile(b"([.,!?\"':;)(])")
, this means use
[ . , ! ? " ' : ; ) ( ]
and space to split the sentence, see
tl.files.basic_tokenizer()
which the default tokenizer of
tl.files.create_vocabulary()
and tl.files.data_to_token_ids()
.
All punctuation marks, such as . , ) (
are all reserved in the vocabularies
of both English and French.
Dataset iteration¶
Loss and update expressions¶
What Next?¶
Cost Functions¶
TensorLayer provides a simple way to creat you own cost function. Take a MLP below for example.
network = tl.InputLayer(x, name='input_layer')
network = tl.DropoutLayer(network, keep=0.8, name='drop1')
network = tl.DenseLayer(network, n_units=800, act = tf.nn.relu, name='relu1')
network = tl.DropoutLayer(network, keep=0.5, name='drop2')
network = tl.DenseLayer(network, n_units=800, act = tf.nn.relu, name='relu2')
network = tl.DropoutLayer(network, keep=0.5, name='drop3')
network = tl.DenseLayer(network, n_units=10, act = identity, name='output_layer')
Regularization of Weights¶
After initializing the variables, the informations of network parameters can be
observed by using network.print_params()
.
sess.run(tf.initialize_all_variables())
network.print_params()
param 0: (784, 800) (mean: -0.000000, median: 0.000004 std: 0.035524)
param 1: (800,) (mean: 0.000000, median: 0.000000 std: 0.000000)
param 2: (800, 800) (mean: 0.000029, median: 0.000031 std: 0.035378)
param 3: (800,) (mean: 0.000000, median: 0.000000 std: 0.000000)
param 4: (800, 10) (mean: 0.000673, median: 0.000763 std: 0.049373)
param 5: (10,) (mean: 0.000000, median: 0.000000 std: 0.000000)
num of params: 1276810
The output of network is network.outputs
, then the cross entropy can be
defined as follow. Besides, to regularize the weights,
the network.all_params
contains all parameters of the network.
In this case, network.all_params = [W1, b1, W2, b2, Wout, bout]
according
to param 0, 1 ... 5 shown by network.print_params()
.
Then max-norm regularization on W1 and W2 can be performed as follow.
y = network.outputs
# Alternatively, you can use tl.cost.cross_entropy(y, y_) instead.
cross_entropy = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(y, y_))
cost = cross_entropy
cost = cost + tl.cost.maxnorm_regularizer(1.0)(network.all_params[0]) +
tl.cost.maxnorm_regularizer(1.0)(network.all_params[2])
In addition, all TensorFlow’s regularizers like
tf.contrib.layers.l2_regularizer
can be used with TensorLayer.
Regularization of Activation outputs¶
Instance method network.print_layers()
prints all outputs of different
layers in order. To achieve regularization on activation output, you can use
network.all_layers
which contains all outputs of different layers.
If you want to apply L1 penalty on the activations of first hidden layer,
just simply add tf.contrib.layers.l2_regularizer(lambda_l1)(network.all_layers[1])
to the cost function.
network.print_layers()
layer 0: Tensor("dropout/mul_1:0", shape=(?, 784), dtype=float32)
layer 1: Tensor("Relu:0", shape=(?, 800), dtype=float32)
layer 2: Tensor("dropout_1/mul_1:0", shape=(?, 800), dtype=float32)
layer 3: Tensor("Relu_1:0", shape=(?, 800), dtype=float32)
layer 4: Tensor("dropout_2/mul_1:0", shape=(?, 800), dtype=float32)
layer 5: Tensor("add_2:0", shape=(?, 10), dtype=float32)
Easy to Modify¶
Modifying Pre-train Behaviour¶
Greedy layer-wise pretrain is an important task for deep neural network initialization, while there are many kinds of pre-train metrics according to different architectures and applications.
For example, the pre-train process of Vanilla Sparse Autoencoder can be implemented by using KL divergence as the following code, but for Deep Rectifier Network, the sparsity can be implemented by using the L1 regularization of activation output.
# Vanilla Sparse Autoencoder
beta = 4
rho = 0.15
p_hat = tf.reduce_mean(activation_out, reduction_indices = 0)
KLD = beta * tf.reduce_sum( rho * tf.log(tf.div(rho, p_hat))
+ (1- rho) * tf.log((1- rho)/ (tf.sub(float(1), p_hat))) )
For this reason, TensorLayer provides a simple way to modify or design your
own pre-train metrice. For Autoencoder, TensorLayer uses ReconLayer.__init__()
to define the reconstruction layer and cost function, to define your own cost
function, just simply modify the self.cost
in ReconLayer.__init__()
.
To creat your own cost expression please read Tensorflow Math.
By default, ReconLayer
only updates the weights and biases of previous 1
layer by using self.train_params = self.all _params[-4:]
, where the 4
parameters are [W_encoder, b_encoder, W_decoder, b_decoder]
. If you want
to update the parameters of previous 2 layers, simply modify [-4:]
to [-6:]
.
ReconLayer.__init__(...):
...
self.train_params = self.all_params[-4:]
...
self.cost = mse + L1_a + L2_w
Adding Customized Layer¶
Contribute useful Layer
as an developer. The source code of TensorLayer is
easy to understand, open tensorlayer/layer.py
and read DenseLayer
, you
can fully understand how it work.
Adding Customized Regularizer¶
See tensorlayer/cost.py
More info¶
For more information on what you can do with TensorLayer, just continue reading through readthedocs. Finally, the reference lists and explains as follow.
layers (tensorlayer.layers
),
weight initializers (tensorlayer.init
),
activation (tensorlayer.activation
),
natural language processing (tensorlayer.nlp
),
reinforcement learning (tensorlayer.rein
),
cost expressions and regularizers (tensorlayer.cost
),
load and save files (tensorlayer.files
),
operating system (tensorlayer.ops
),
helper functions (tensorlayer.utils
),
visualization (tensorlayer.visualize
),
iteration functions (tensorlayer.iterate
),
preprocessing functions (tensorlayer.preprocess
),


@(implicit request: Request[Any], environment: play.api.Environment, code: java.util.UUID)
@main("reset password",None,"reset passord",false) {
} {
} {
Reset Password
}
@*
*
* Copyright (c) 2015-2020 Rodney S.K. Lai
* https://github.com/rodney-lai
*
* Permission to use, copy, modify, and/or distribute this software for
* any purpose with or without fee is hereby granted, provided that the
* above copyright notice and this permission notice appear in all copies.
 IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
* ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
* WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
* ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
* OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*
*@


Edinson Cavani plays down 'fight offer' to Lionel Messi after heated bust-up
Edinson Cavani has played down his spat with Lionel Messi after claims the striker asked the Argentina captain if he ‘wanted to fight’.
Argentina and Uruguay played out a heated 2-2 draw in Tel Aviv on Monday night, which saw Cavani and Messi clash on several occasions.
An argument broke out in the first half following a strong tackle and Cavani reportedly invited Messi to settle their differences.
Messi replied to the fight offer with ‘whenever you want’ and the pair needed to be separated in the tunnel by Barcelona and Uruguay striker Luiz Suarez as tempers continued to boil over off the pitch.
Cavani has now spoken about the incident, insisting it was all part of the rivalry between Argentine and Uruguay.
‘It’s part of football, Clasicos are like this,’ he said after the game.
Advertisement Advertisement
‘The truth is that it was at times a game that had friction, there were challenges and things that are part of football, especially of South American [football].
‘We take positives from our performance and the fact that we were able to score two goals. We still need to improve but it’s good to end the year like this.’
Uruguay twice led against Argentina through Cavani and Suarez, before Messi scored a 90th-minute free-kick to rescue a draw for his side in the friendly.
It was just the second time Messi featured for Argentina since serving a three-month ban for claiming the Copa America was rigged in Brazil’s favour.


The vibrating equipment is an efficient new, multi-layer vibration equipment for stone quarries and sand filters. Vibrating screen designed by our company is a kind of screening equipment, in other word, vibrating screener is a machine tool to separate aggregates by particle size by means of an exciting force applied to the machine causing it to vibrate. Zenith's vibrating screen can be divided into the following types: inclined screen, horizontal screen, YK series vibrating screen, triple-shafted horizontal screen and high speed screen.
1, the use of tilt angle structure, small amplitude, high frequency, high efficiency of the device screen.
2, excitation power, the exciting force by block eccentric.
3, the use of high strength bolts with a sieve screen box beams will be combined effective and smooth running.
4,Simple and dependable Eccentric type system; High screening capacity;High durability;No transmission of screen panels;Ready changing of screen plates;Rigid and vibrating resistance screen body.
Mobile vibrating screen is widely applied to screen dry materials in ore dressing, building materials, concrete, rock, crushed stone, quarry waste, ore mineral mining, construction, demolition debris, aggregate, dewatering industrial, sand and gravel etc. The advantaged of this series screen are as follows: stable operation, lower energy consumption, low noise, high efficiency and long service life.
Zenith is one of the world top vibrating screen and other beneficiation equipment suppliers. Our mobile vibrating screen are extensively applied in mining and construction sector. It plays a vital function in quarry and mining process of various ores and stones. Zenith's crushing plant, grinding mills, sand making machine, sand washer, vibrating feeder and screen have been exported to a lot of nations including South Africa, India, Brazil, Indonesia, Kenya, Philippines and so forth. If you would like to know vibrating screen design and style and specification details or other info, please feel free to contact us. We will offer sincere service to you.
Copyright © 2007-2014 MiningTech
Confidentiality of customers' information is our utmost concern


The book What Hath God Wrought: The Transformation of America, 1815 - 1848 describes a shift in the physics of information.
This book highlights a story published by a British paper shortly after the telegraph was up and running. The story describes a man killing his wife, and boarding a train before the information of his crime spread. A description of the man was telegraphed ahead of the train, and the man was caught.
The fact that authorities were able to catch the man isn't very amazing to us. But, for people in the mid-1800s, information traveling faster than a train or horse or boat was new. Before the mid-1800s, human digestible information, literally, could not travel faster than a train.
The book highlights a few other stories like this. Andrew Jackson's famous Battle of New Orleans took place after the British surrendered (it takes a long time to send a message by boat).
I love the idea of overcoming old physical limitations.
I also enjoy thinking of information in a physical light.
How does it spread?
What's its fundamental constituents?
And the like..
I'm looking forward to the 'TBA' Information Theory course on coursera


A novel implementation that solves the linear distance weighted discrimination and the kernel distance weighted discrimination.
| Version: | 2.0.2 |
| Depends: | methods |
| Imports: | graphics, grDevices, stats, utils |
| Published: | 2018-04-21 |
| Author: | Boxiang Wang, Hui Zou |
| Maintainer: | Boxiang Wang <boxiang at umn.edu> |
| License: | GPL-2 |
| NeedsCompilation: | yes |
| CRAN checks: | kerndwd results |
| Reference manual: | kerndwd.pdf |
| Package source: | kerndwd_2.0.2.tar.gz |
| Windows binaries: | r-devel: kerndwd_2.0.2.zip, r-release: kerndwd_2.0.2.zip, r-oldrel: kerndwd_2.0.2.zip |
| OS X binaries: | r-release: kerndwd_2.0.2.tgz, r-oldrel: kerndwd_2.0.2.tgz |
| Old sources: | kerndwd archive |
Please use the canonical form https://CRAN.R-project.org/package=kerndwd to link to this page.


The -addheader command adds RAMADDA's point data header to the output file. The header specifies the canonical field ids and a set of properties for each field and is of the form:
#fields=field1[prop1=value1 prop2=value2 ...],field2[],...,fieldN[propn=valuen ...] v1,v2,v3 ...The addheader command is run as below passing in one set of name/value arguments. If the value has spaces then group the text with {...} brackets. seesv:-addheader "arg1 value1 arg2 {value 2} arg3 value3" The args are typically of the form canonical_field_name.some_property where canonical field name is the column header cleaned up - lowercase, spaces converted to "_", strip off non alphanumeric characters.
For example, say you have a file like this: some date,year,some field 2018-09-01,2023,10 2019-10-01,2024,20 2020-09-01,2024,30 Running this command: seesv: -addheader "some_date.type date date.format yyyy-MM-dd year.type date year.format yyyy some_field.id temperature some_field.label {The Temperature} some_field.type celsius" -p testdate.csv Will result in an output:
#fields=some_date[label="Some Date" type="date" format="yyyy-MM-dd" ] ,year[label="Year" type="date" format="yyyy" ] ,temperature[label="Some Field" type="integer" chartable="true" ] 2018-09-01,2023,10 2019-10-01,2024,20This specifies:
value1,value2,value3 10,20,30You can specify the type as "value.*.type integer".
Adding a grouping to the pattern can be used to pull out text from the field name and use it in the value. For example, below would set the label: seesv: -addheader "(value.*).*.label {Some label \$1}"


| Apache Struts 2 Documentation > Home > Guides > User Guide > Using Freemarker Templates |
FreeMarker is a Java-based template engine that is a great alternative to JSP. FreeMarker is ideal for situations where your action results can possibly be loaded from outside a Servlet container. For example, if you wished to support plugins in your application, you might wish to use FreeMarker so that the plugins could provide the entire action class and view in a single jar that is loaded from the classloader.
Struts provides a result type "freemarker" which renders a FreeMarker template. This result type is defined in struts-default.xml. To create pages using FreeMarker, set the result type of the actions to "freemarker".
<include file="struts-default.xml"/> ... <action name="test" class="package.Test"> <result type="freemarker">/WEB-INF/views/testView.ftl</result> </action> ...
FreeMarker uses the ${...} notation to access properties. They are called interpolations. Properties on the actions (getter methods) will automatically be available on the FreeMarker templates. If an action has a "getName()" method, then its value can be inserted on the template like:
Your name is: ${name}
| Be Careful By default, FreeMarker will throw an error if it finds a variable that is not defined, or has a null value. See this FAQ for details. |
The following are ways to obtained Application scope attributes, Session scope attributes, Request scope attributes, Request parameters, and framework Context scope parameters:-
Assuming there's an attribute with name myApplicationAttribute in the Application scope.
<#if Application.myApplicationAttribute?exists> ${Application.myApplicationAttribute} </#if>
or
<EMAIL_ADDRESS>value="%{#application.myApplicationAttribute}" />
Assuming there's an attribute with name mySessionAttribute in the Session scope.
<#if Session.mySessionAttribute?exists> ${Session.mySessionAttribute} </#if>
or
<EMAIL_ADDRESS>value="%{#session.mySessionAttribute}" />
Assuming there's an attribute with name 'myRequestAttribute' in the Request scope.
<#if Request.myRequestAttribute?exists> ${Request.myRequestAttribute} </#if>
or
<EMAIL_ADDRESS>value="%{#request.myRequestAttribute}" />
Assuming there's a request parameter myParameter (eg. http://host/myApp/myAction.action?myParameter=one).
<#if Parameters.myParameter?exists> ${Parameters.myParameter} </#if>
or
<EMAIL_ADDRESS>value="%{#parameters.myParameter}" />
Assuming there's a parameter with the name myContextParam in framework context.
${stack.findValue('#myContextParam')}
or
<EMAIL_ADDRESS>value="%{#myContextParam}" />
The framework looks for FreeMarker templates in two locations (in this order):
This ordering makes it ideal for providing templates inside a fully-built jar, but allowing for overrides of those templates to be defined in your web application. In fact, this is how you can override the default UI tags and Form Tags included with the framework.
In addition, you can specify a location (directory on your file system) through the templatePath or TemplatePath context variable (in the web.xml. If a variable is specified, the content of the directory it points to will be searched first.
| This variable is currently NOT relative to the root of your application. |
If a property is defined on the template with the same name as a property on the action, FreeMarker will use the property defined on the template.
When using FreeMarker with the framework, variables are looked up in several different places, in this order:
Note that the action context is looked up after the value stack. This means that you can reference the variable without the typical preceding has marker (#) like you would have to when using the JSP s:property tag. This is a nice convenience, though be careful because there is a small chance it could trip you up.
<EMAIL_ADDRESS>id="url" value="http://www.yahoo.com"/> Click here!
The built-in variables that Struts-FreeMarker integration provides are:
| Name | Description |
|---|---|
| stack | The value stack itself, useful for calls like ${stack.findString('ognl expr')} |
| action | The action most recently executed |
| response | The HttpServletResponse |
| res | Same as response |
| request | The HttpServletRequest |
| req | Same as request |
| session | The HttpSession |
| application | The ServletContext |
| base | The request's context path |
To configure the FreeMarker engine, just add a file freemarker.properties to the classpath. The supported properties are those that the FreeMarker Configuration object expects, see FreeMarker's documentation for more details.
default_encoding=ISO-8859-1 template_update_delay=5 locale=no_NO
Tags distributed with Struts are automatically made available to FreeMarker templates. To use any tag add<EMAIL_ADDRESS>in front of the tag name. Like:
<EMAIL_ADDRESS>test="printName"><EMAIL_ADDRESS>value="myBeanProperty" /><EMAIL_ADDRESS>To use JSP tags that are not part of Struts you have to:
1. Add JspSupportSerlvet to web.xml
<servlet> <servlet-name>JspSupportServlet</servlet-name> <servlet-class>org.apache.struts2.views.JspSupportServlet</servlet-class> <load-on-startup>1</load-on-startup> </servlet>
2. Declare the tld on web.xml or use FreeMarker's "assign" directive. When using the "assign" directive, provide the absolute path to the tld file.
<#assign ex=JspTaglibs["/WEB-INF/example.tld"] /><EMAIL_ADDRESS>text="hello" />
There are some advanced features that may be useful when building Struts applications with FreeMarker.
FreeMarker has built in support for formatting dates and numbers. The formatting rules are based on the locale associated with the action request, which is by default set in struts.properties but can be over-ridden using the I18n Interceptor. This is normally perfect for your needs, but it is important to remember that these formatting rules are handled by FreeMarker and not by the framework's Type Conversion support.
If you want the framework to handle the formatting according to the Type Conversion you have specified, you shouldn't use the normal ${...} syntax. Instead, you should use the property tag. The difference is that the property tag is specifically designed to take an OGNL expression, evaluate it, and then convert it to a String using any Type Conversion rules you have specified. The normal ${...} syntax will use a FreeMarker expression language, evaluate it, and then convert it to a String using the built in formatting rules.
| The difference in how type conversion is handled under Freemarker is subtle but important to understand. |
In FreeMarker it is incorrect to quote non string values. If a value is quoted, then an string will be passed, instead of the expected object, causing an exception. For example, the "textarea" tag expects the attributes "rows" and "cols" of type Integer:
<EMAIL_ADDRESS>rows=5 cols=40 />
Sometimes you may with to extend the framework's FreeMarker support. For example, you might want to extend the Struts tags that come bundled with the framework.
To extend the Freemarker support, develop a class that extends org.apache.struts2.views.freemarker.FreemarkerManager, overriding methods as needed, and plugin the class through the struts.properties:
struts.freemarker.manager.classname = com.yourcompany.YourFreeMarkerManager
FreeMarker by default uses the "<#directive />" syntax. FreeMarker supports an alternative syntax, where [ and ] are used instead of < and >. To enable the alternative syntax, add ftl at the beginning of the template. The alternative syntax makes it easier to differentiate between FreeMarker directives, and JSP or HTML tags.
[#ftl]  FreeMarker Example
Alternative Syntax
<EMAIL_ADDRESS>test="printName"]<EMAIL_ADDRESS>value="myBeanProperty" /]<EMAIL_ADDRESS>


Warning! You have no authority to access this page! Please log in or sign up to WikiNote. OR Contact to administrator for acquiring the authority. ID Password


8/30/88
These maps include all of the areas in the colony base that I could find.
North is at the top of the maps. A legend for the door markings is found on the first level map. Doors that stand off from the walls go through a space warp to another location. The destinations are not marked on these doors. (I gotta leave some of the work for you!) The entrance to the tunnel to the teleporter labs is off lab two on level 4.
Connections in the 3x3 maze in level 5 are not shown, since the order of the rooms appears to change while you're in the maze. The 5x5 structure there is my theory, but it seemed logical. If you head east in the 3x3 maze, you have a fair chance of exiting the maze. However, one of the rooms in the 3x3 maze has no exits.
Enjoy! -- Pat Gratton


{% extends "base_generic.html" %} {% block content %}
Name: {{ player.player_name }} || Seat at {{ player.id }}
There is no vision! Something went wrong!


© 2019
| subacaulis subacaulis subacaule | almost stemless | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| a | a | pre | α- not, without, less | |
| caulis | caul | adj/ | καυλοϛ stem of a plant; usual spelling was colis or coles | |
| Pediomelum subacaule (Torr. & Gray) Rydb. Camissonia subacaulis (Pursh) Raven | ||||
| subalbicans subalbicans subalbicans | less than white | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| albicans | albicant | apar | albico to be white | |
| Leprocaulon subalbicans (Lamb) Lamb & Ward | ||||
| subalpinus subalpina subalpinum | from lower mountain heights | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| alpinus | alpin | adj | of alps, mountains | |
| Draba subalpina Goodman & C.L.Hitchc. Eriogonum subalpinum Greene Calochortus subalpinus Piper | ||||
| subasperrimus subasperrima subasperrimum | less than roughest | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| superlative of asper | asper | adj | rough | |
| rimus | rim | adj | superlative suffix; most so, to the greatest degree; most-, -est (e.g. largest, prettiest, whitest) | |
| Brachythecium subasperrimum Card. & Thăr. | ||||
| subbiflorus subbiflora subbiflorum | almost 2 flowered | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| bi | bi | pre | bis twice | |
| florus | flor | ppar | floreo to bloom, to flower | |
| Lotus subbiflorus Lag. | ||||
| subcapitatus subcapitata subcapitatum | somewhat with heads | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| capitatus | capitat | adj | having a head | |
| Draba subcapitata Simm. | ||||
| subcarnosus subcarnosa subcarnosum | somewhat fleshy | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| carnosus | carnos | adj | fleshy | |
| Lupinus subcarnosus Hook. | ||||
| subcaulescens subcaulescens subcaulescens | short stem | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| caulis | caul | noun/m | καυλοϛ stem of a plant; usual spelling was colis or coles | |
| escens | escent | adj | adjective suffix for nouns or adjectives: becoming like, incomplete resemblance -ish | |
| Selaginella subcaulescens Baker | ||||
| subciliatus subciliata subciliatum | somewhat hairy | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| cilium | cili | noun/n | eyelash & eyelid together | |
| atus | at | adj | adjectival suffix for nouns: possessive of or likeness of something (with, shaped, made)/ for verb participles: a completed action, -ed | |
| Silene subciliata B.L.Robins. | ||||
| subdecoloratus subdecolorata subdecoloratum | somewhat discolored | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| decolor | decolor | adj | without the natural color | |
| atus | at | adj | adjectival suffix for nouns: possessive of or likeness of something (with, shaped, made)/ for verb participles: a completed action, -ed | |
| suberosus suberosa suberosum | with corklike features | |||
| suber | suber | noun/n | the cork tree: Quercus suber; cork | |
| osus | os | adj | adjective suffix for nouns: plenitude or notable development | |
| Passiflora suberosa L. Rhododendron suberosum Balf. f. & Forrest | ||||
| subflabellatus subflabellata subflabellatum | somewhat fanlike | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| flabellum | flabell | noun/n | small hand fan | |
| atus | at | adj | adjectival suffix for nouns: possessive of or likeness of something (with, shaped, made)/ for verb participles: a completed action, -ed | |
| Aciphylla subflabellata W.Oliv. | ||||
| subgemmifer subgemmifera subgemmiferum | bearing fewer 'gems' | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| gemma | gemm | noun/f | originally, a bud on a plant; a gem | |
| i | i | cnct | connective vowel used by botanical Latin | |
| fer | fer | apar | fero to bear, carry, bring | |
| subgibbosus subgibbosa subgibbosum | somewhat humped | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| gibbus | gibb | noun/m | hump, hunch | |
| osus | os | adj | adjective suffix for nouns: plenitude or notable development | |
| Neoporteria subgibbosa (Haw.) Britton & Rose | ||||
| subglaber subglabra subglabrum | approaching hairless | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| glaber | glabr | adj | hairless, smooth, bald | |
| Penstemon subglaber Rydb. | ||||
| subglobosus subglobosa subglobosum | almost spherical | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| globosus | globos | adj | spherical | |
| Livistona subglobosa (Hassk.) C. Mart. Anthracothecium subglobosum Riddle Saribus subglobosus Hassk. | ||||
| subhirsutus subhirsuta subhirsutum | somewhat hairy | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| hirsutus | hirsut | adj | rough, shaggy, bristly | |
| Allium subhirsutum L. Lupinus subhirsutus A. Davids. | ||||
| subhirtellus subhirtella subhirtellum | almost a little hairy | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| hirtus | hirt | adj | hairy, shaggy, rough, uncultivated | |
| ellus | ell | adj | adjective suffix for adjective: diminutive | |
| Prunus subhirtella Miq. | ||||
| subincisus subincisa subincisum | somewhat incised | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| incisus | incis | ppar | incido to cut into, through, up | |
| Ctenitis subincisa (Willd.) Ching | ||||
| subinteger subintegra subintegrum | mostly entire | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| integer | integr | adj | entire, whole, complete; unbroken, unhurt; fresh, new | |
| Hymenoxys subintegra Cockerell | ||||
| subintegerrimus subintegerrima subintegerrimum | most entire | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| superlative of integer | integer | adj | entire, whole, complete; unbroken, unhurt; fresh, new | |
| rimus | rim | adj | superlative suffix; most so, to the greatest degree; most-, -est (e.g. largest, prettiest, whitest) | |
| Aster subintegerrimus (Trautv.) Ostenf. & Resvoll | ||||
| sublimis sublimis sublime | high or lofty | |||
| sublimis | sublim | adj | high, lofty, exaulted | |
| submammillaris submammillaris submammillare | small nipples | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| mammilla | mammill | noun/f | incorrect spelling of mamilla; a breast, teat | |
| aris | ar | adj | alis adjective suffix for nouns: of or pertaining to | |
| submammulosus submammulosa submammulosum | small nippled | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| mammula | mammul | noun/f | a small breast, teat | |
| osus | os | adj | adjective suffix for nouns: plenitude or notable development | |
| submarginatus submarginata submarginatum | small margined | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| marginatus | marginat | ppar | marginato to furnish with a border; to border, enclose with a margin | |
| Abrotanella submarginata A.Gray | ||||
| submersus submersa submersum | under water | |||
| submersus | submers | ppar | summergo to sink, submerge (also submergo) | |
| Aspicilia submersa (Lamy) Hue Ceratophyllum submersum L. | ||||
| subminutissimus subminutissima subminutissimum | smaller than the smallest | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| superlative of minutus | minut | ppar | small, minute, little | |
| issimus | issim | adj | superlative suffix; most so, to the greatest degree; most-, -est (e.g. largest, prettiest, whitest) | |
| Arthonia subminutissima Nyl. | ||||
| subnudus subnuda subnudum | somewhat bare | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| nudus | nud | adj | naked, bare, unclothed | |
| Gilia subnuda Torr. ex Gray Thelesperma subnudum Gray | ||||
| subpumilus subpumila subpumilum | somewhat dwarfed | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| pumilus | pumil | adj | dwarfish, little | |
| subpuniceus subpunicea subpuniceum | somewhat purple | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| puniceus | punice | adj | reddish, purple | |
| Haematomma subpuniceum (Mull. Arg.) de Lesd. | ||||
| subradiatus subradiata subradiatum | paritally rayed | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| radiatus | radiat | ppar | with rays; shining | |
| Cladonia subradiata (Vainio) Sandst. Placynthium subradiatum (Nyl.) Arnold | ||||
| subramosus subramosa subramosum | less than 'many branched' | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| ramosus | ramos | adj | with many branches | |
| Lecidea subramosa Lowe | ||||
| subrigidus subrigida subrigidum | less rigid | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| rigidus | rigid | adj | stiff, hard, inflexible, rigid | |
| Echeveria subrigida (B. L. Rob. & Seaton) Rose | ||||
| subrotundifolius subrotundifolia subrotundifolium | leaves less round | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| rotundus | rotund | adj | round, spherical | |
| i | i | cnct | connective vowel used by botanical Latin | |
| folius | foli | adj | folium leaf | |
| Crataegus subrotundifolia Sarg. | ||||
| subrotundus subrotunda subrotundum | less round | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| rotundus | rotund | adj | round, spherical | |
| Osteomeles subrotunda K. Koch Cinclidium subrotundum Lindb. | ||||
| subrubellus subrubella subrubellum | somewhat reddish | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| rubellus | rubell | adj | reddish | |
| Arthonia subrubella Nyl. | ||||
| subrugosus subrugosa subrugosum | less wrinkled | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| rugosus | rugos | adj | wrinkled, shriveled | |
| Lecanora subrugosa Nyl. | ||||
| subsaxatilis subsaxatilis subsaxatile | growing among fewer rocks | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| saxatilis | saxatil | adj | dwells or is found among rocks, frequents rocks | |
| Hypotrachyna subsaxatilis (de Lesd.) Hale | ||||
| subscaposus subscaposa subscaposum | under scape | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| scaposus | scapos | adj | scapus shaft, stem, stalk, trunk | |
| Brassica subscaposa Maire & Weiller | ||||
| subsecundus subsecunda subsecundum | partially secund | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| secundus | secund | adj | following, next, second | |
| Sphagnum subsecundum Nees in Sturm Elymus subsecundus (Link) A. Love & D. Love | ||||
| subsericeus subsericea subsericeum | partially silken | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| sericeus | serice | adj | sericis σερικοϛ of silk (misspelled: sericeus) | |
| Raoulia subsericea Hook. f. Ceanothus subsericeus Rydb. | ||||
| subserratus subserrata subserratum | partially serrated | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| serratus | serrat | adj | saw like, serrated, saw toothed | |
| Arabis subserrata Greene Penstemon subserratus Pennell | ||||
| subsessilis subsessilis subsessile | nearly sessil | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| sessilis | sessil | adj | seated; (of flowers) low, dwarf | |
| Rhododendron subsessile Rendle Aster subsessilis Burgess | ||||
| subsimilis subsimilis subsimile | somewhat similar | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| similis | simil | adj | like, resembling, similar | |
| subspicatus subspicata subspicatum | mostly spiked | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| spicatus | spicat | ppar | spico to grow ears, spikes (like corn) | |
| Sansevieria subspicata Baker Symphyotrichum subspicatum (Nees) Nesom Aster subspicatus Nees | ||||
| subspinosus subspinosa subspinosum | somewhat spiny | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| spinosus | spinos | adj | thorny | |
| Saurauia subspinosa J. Anthony Teucrium subspinosum Pourr. ex Willd. | ||||
| subterraneus subterranea subterraneum | underground | |||
| subterraneus | subterrane | adj | subterranean, underground | |
| Glycine subterranea L. Trifolium subterraneum L. Turbinicarpus subterraneus (Backeb.) A. D. Zimm. | ||||
| subtilissimus subtilissima subtilissimum | most delicate | |||
| superlative of subtilis | subtil | adj | fine, thin, delicate | |
| issimus | issim | adj | superlative suffix; most so, to the greatest degree; most-, -est (e.g. largest, prettiest, whitest) | |
| Allium subtilissimum Ledeb. | ||||
| subtrinervis subtrinervis subtrinerve | mostly 3 veined | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| tri | tri | pre | tres three | |
| nervus | nerv | noun/m | νευρον a sinew, tendon, nerve | |
| is | adj | adjectival suffix | ||
| Erigeron subtrinervis Rydb. ex Porter & Britt. | ||||
| subtrinervius subtrinervia subtrinervium | barely three nerved | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| tri | tri | pre | tres three | |
| nervus | nerv | noun/m | νευρον a sinew, tendon, nerve | |
| ius | i | adj | -ιοϛ adjective suffix for nouns: characteristic of, connected to, resembling, when related to -eus: of..., ...like | |
| Acer subtrinervium F. P. Metcalf | ||||
| subtriplinervius subtriplinervia subtriplinervium | mostly 3 veined | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| triplus | tripl | adj | τριπλουϛ triple, threefold | |
| i | i | cnct | connective vowel used by botanical Latin | |
| nervus | nerv | noun/m | νευρον a sinew, tendon, nerve | |
| ius | i | adj | -ιοϛ adjective suffix for nouns: characteristic of, connected to, resembling, when related to -eus: of..., ...like | |
| Ficus subtriplinervia C. Mart. Heterocentron subtriplinervium (Link & Otto) A.Braun & Bouche | ||||
| subulatus subulata subulatum | awl shaped | |||
| subula | subul | noun/f | awl | |
| atus | at | adj | adjectival suffix for nouns: possessive of or likeness of something (with, shaped, made)/ for verb participles: a completed action, -ed | |
| Phlox subulata L. Phleum subulatum (Savi) Asch. & Graebn. Aster subulatus Michx. | ||||
| subvestitus subvestita subvestitum | somewhat clothed | |||
| sub | sub | pre | below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| vestitus | vestit | ppar | vestio to clothe, to dress | |
| Ribes subvestitum Hook. & Arn. Astragalus subvestitus (Jepson) Barneby | ||||
| succulentus succulenta succulentum | markedly juicy | |||
| succus | succ | noun/m | sucus juice or moisture of living things; strength, vigor | |
| ulentus | ulent | adj | adjective suffix for nouns: plenitude or notable development, from more recent coinage of words, see '-osus' | |
| Bourreria succulenta Jacq. Crossidium succulentum Holz. & Bartr. Lupinus succulentus Dougl. ex K.Koch | ||||
| succumbens succumbens succumbens | falling down | |||
| succumbens | succumbent | apar | succumbo to fall, lie, or sink down | |
| Astragalus succumbens Dougl. ex Hook. | ||||
| suffocatus suffocata suffocatum | strangled | |||
| suffocatus | suffocat | ppar | suffoco to strangle, suffocate, choke | |
| Trifolium suffocatum L. | ||||
| suffrutescens suffrutescens suffrutescens | somewhat shrubby | |||
| suf | suf | pre | sub below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| frutescens | frutescent | apar | frutico to put out shoots, to become bushy | |
| Antennaria suffrutescens Greene | ||||
| suffruticosus suffruticosa suffruticosum | somewhat shrubby | |||
| suf | suf | pre | sub below, under, up from under, almost, approaching; botany: less so than a similar plant | |
| fruticosus | fruticos | adj | bushy, shrubby | |
| Calendula suffruticosa Vahl Epilobium suffruticosum Nutt. Haplopappus suffruticosus (Nutt.) Gray | ||||
| sulcatus sulcata sulcatum | furrowed | |||
| sulcatus | sulcat | ppar | sulco to plow, furrow | |
| Sehima sulcata (Hack.) A. Camus Trillium sulcatum Patrick Anthoceros sulcatus Austin | ||||
| sulcinervius sulcinervia sulcinervium | veins impressed | |||
| sulcus | sulc | noun/m | furrow, ditch | |
| i | i | cnct | connective vowel used by botanical Latin | |
| nervus | nerv | noun/m | νευρον a sinew, tendon, nerve | |
| ius | i | adj | -ιοϛ adjective suffix for nouns: characteristic of, connected to, resembling, when related to -eus: of..., ...like | |
| Huperzia sulcinervia (Spring) Trevisan Lycopodium sulcinervium Spring Urostachys sulcinervius (Spring) Herter ex Nessel | ||||
| sulphureus sulphurea sulphureum | yellow like sulphur | |||
| sulphureus | sulphure | adj | sulpherous (also sulfureus) | |
| Castilleja sulphurea Rydb. Bryophyllum sulphureum (Baker) A. Berger Cosmos sulphureus Cav. | ||||
| superbus superba superbum | superb | |||
| superbus | superb | adj | haughty, arrogant; superior, splendid | |
| Gloriosa superba L. Lilium superbum L. Calochortus superbus Purdy ex J.T.Howell | ||||
| supinus supina supinum | lying down | |||
| supinus | supin | adj | bent backwards, lying on the back | |
| Linaria supina (L.) Chaz. Gnaphalium supinum L. Juncus supinus Moench | ||||
| surculosus surculosa surculosum | suckering | |||
| surculus | surcul | adj | a young branch, sprout; a graft, sucker | |
| osus | os | adj | adjective suffix for nouns: plenitude or notable development | |
| Eurybia surculosa (Michx.) Nesom Elatostema surculosum Wight Aster surculosus Michx. | ||||
| suspensus suspensa suspensum | hanging | |||
| suspensus | suspens | ppar | suspendo to hand, suspend | |
| Forsythia suspensa (Thunb.) Vahl Viburnum suspensum Lindley | ||||
| sylvaticus sylvatica sylvaticum | growing in the woods, wild | |||
| sylva | sylv | noun/f | silva woods, forest | |
| aticus | atic | adj | adjective suffix for nouns: habitat | |
| Carex sylvatica Huds. Geranium sylvaticum L. Scirpus sylvaticus L. p.p. | ||||
| sylvester sylvestris sylvestre | growing in the woods, wild | |||
| sylvester | sylvestr | adj | sylvestris of or belonging to the forest or woods, more correctly: silvestris, sometimes silvester | |
| Aruncus sylvester Kostel. ex Maxim. Chaerophyllum sylvestre L. | ||||
| sylvestris sylvestris sylvestre | growing in the woods, wild | |||
| sylvestris | sylvestr | adj | of or belonging to the forest or woods, more correctly: silvestris, sometimes silvester | |
| Althaea sylvestris Briganti |


circos_plot.Rd
Creates a circos plot from the list of ligands and receptors
circos_plot( ligand_receptor_frame, cell_group_colors, ligand_color = "blue", receptor_color = "red", cex_outer = 0.5, cex_inner = 0.4 )
| ligand_receptor_frame | Resulting tibble (usually filtered in some way) from the celltalk function. |
|---|---|
| cell_group_colors | Colors used for the groups of cells in the outer track of the circos plot. |
| ligand_color | Color to use for ligands. Defaults to "blue". |
| receptor_color | Color to use for the receptors. Defaults to "red". |
| cex_outer | Size of the text for the cell groups in the outer layer of the circos plot. Default is 0.5. |
| cex_inner | Size of the text for the ligand and receptors in the inner layer of the circos plot. Default is 0.4. |
Generates a circos plot connecting ligands and receptors across cell types for a given sample group


Changelog
Latest
Version 2020.3
May 31, 2020
- 🎈New: Custom Fonts Support
- 🎈New: “Prettify JSON”
- 🎈New: “Prettify and Sort JSON”
- ⚙Improved: “List: Add Line Numbers\Bullets” skips empty lines
- 🐛Fixed: Dark theme for Action Extension
- 🐛Fixed: Long function’s description now fit on screen
Version 2020.1
Feb 28, 2020
- 🎈New “Wrap Paragraph in Quotes”
- 🎈New “Text Stats”
- 🎈New “List: Add Line Numbers \ Bullets” functions, 4 variants
- 🎈Dynamic Text support
- 🐛Fixed a bunch of typos
Version 2.1
Dec 29, 2019
- 🎈New “Spell out Numbers” function.
- 🎈New “Strip Numeric Characters” function.
- 🎈New “Remove All non Alphanumeric Characters” function.
- 🎈New “What’s New” section in settings.
- 🎈New “Done” button to dismiss the keyboard
- 🐛Fixed: Today Widget is unreadable when system is in Dark Mode
Version 2.0.2-3
Oct 10, 2019
- 🎈New Unquote Sentence Functions. Unquotes sentence ignoring quotes within the sentence.
- 🐛Fixed: Context menus style(Dark\Light) may not match app style in certain conditions.
Version 2.0.1
Sep 22, 2019
- 🐛Fixes for iPhone layout
Version 2.0
Sep 19, 2019
This major update of Esse brings new functions, new iPad optimized layout and full support of the latest iOS, including new powerful Shortcuts.
- 🎈New Scratchpad. Transform your text by combining multiple functions at once. Optionally you can save a set of functions as a custom function to use it later.
- 🎈New shortcuts with natural language style parameters will simplify the use of Esse functions in the Shortcuts app. A brand new “Transform Text File” shortcut action can apply Esse functions to a text file, stored in iCloud Drive
- 🎈New iPad optimized interface layout, including new multi column view and context menus
- 🎈New Count Unique Words function
- 🎈New Extract Phone Number function
- 🎈New Extract Email function
- 🎈New Extract Date function
- 🎈New Extract Address function
- 🎈New Chicago Manual of Style capitalization function
- 🎈New Shuffle Words function
- 🎈New Shuffle Sentences function
- ⚙Minor Search improvements, now you can search functions not only by name but by description as well
- 🎈Support for Light and Dark theme, with option to override System appearance
- 🐛Fixed: Custom functions not visible in Share sheet and Today widget
Esse stays free of charge, ads and any kind of tracking. We hope it can make your life a little easier, if you like the app - spread a word.
1.3
May 11, 2019
- 🎈New “Extract URL” - Extracts URLs from given text, outputs one URL per line
- ⚙Updated “Wrap in Quotes” - Wraps provided text in quotes, replacing all existing double quotes with single quotes
- 🐛Fixed: Custom functions not showing in Action Extension and Today Widget more
1.2
Feb 5, 2019
Features:
- 🎈New function: ROT13 Conversion
- ⚙Share Button to quickly share text conversion result
- ⚙Transformation preview on function page live updated with the current input text
- ⚙Keyboard shortcuts: CMD+V to paste from clipboard, CMD+F to start search
- ⚙All Icons now follow theme accent color
- 🐛Fixed: Keyboard appearance doesn’t follow current theme
- 🐛Fixed: UI glitch(wrong color) in Function Library after theme switch
1.1
Dec 17, 2018
Two new transformations:
- 🎈Emojify - automatically replace text with emojis
- 🎈Strip non numeric characters - handy when you want to unify phone numbers, i.e. (234) 222-112-33 -><PHONE_NUMBER>3
- ⚙UI has been optimized to support new iPads Pro.
1.0
Oct 13, 2018
Initial release.


YUI 3.10.2 Released
We are happy to announce the release of YUI 3.10.2! You can find it now on the Yahoo! CDN, download it directly, or pull it in via npm. We’ve also updated the YUI Library website with the latest documentation.
{% block style %} {% endblock %}
{% block body %}We are happy to announce the release of YUI 3.10.2! You can find it now on the Yahoo! CDN, download it directly, or pull it in via npm. We’ve also updated the YUI Library website with the latest documentation.