2.4.5
Newbuilding supervision by owners
Owners were represented during the building phase of VIKING SALLY by a total of 10 persons, although not simultaneously. These were
| Alf Johansson | - managing director |
| Sven Erik Johansson | - managing director |
| Yngve Röblom | - technical superintendent |
| Nils Brunström | - nautical superintendent |
| later replaced by | |
| Lars Mäkki | - the subsequent master of the vessel |
| Bertil Hansson | - administration manager |
| Lars Karlsson | - chief engineer |
| Stig Strömberg | - 1st engineer |
| Stig Lindström | - chief officer |
| Lars Sjögren | - electrician |
In addition, there were 2 architects frequently at the Yard:
| Pekka Perjo | - internal architect |
| Heikki Sorvali | - external architect |
Chief engineer Lars Karlsson arrived already at Papenburg three days after the contract was signed together with Captain Brunström (who was replaced five weeks later by Lars Mäkki, the subsequent master of VIKING SALLY), 1st engineer Stig Strömberg and electrician Lars Sjögren. He had been in Papenburg already as owners' representative during the building of VIKING 3, 4 & 5, thus knew the Yard as well as the people including the B.V. inspectors (all vessels were classed by B.V.), spoke some German and was therefore nominated as "coordinator" of owners' superintendents. He stayed until the delivery and subsequently became chief engineer on VIKING SALLY. Apart from the usual leave periods he remained in that position until 30.03.92.
The following is an abstract of his statement concerning the building phase:
«Three days after signing of the contract I came to Papenburg together with Capt. Brunström (was relieved after 5 weeks by Lars Mäkki), 1st engineer Stig Strömberg, and electrician Lars Sjögren. At the beginning I had only 1 small GA plan, deck and engine specifications were shown to me.
Due to pressure of time we had to make a lot of compromises. In my opinion and experience the Meyer Werft was the only yard in Europe able to do such a job because of so little bureaucracy. This has, in my opinion, nothing to do with the catastrophe. Meyer Werft has very much experience. They have built all ships very strong, i.e. they were usually heavy. It is a well known fact that Meyer ships were stronger and behaved better in ice compared to e.g. Wärtsilä ships.
With my team of supervisors I checked everything including, and in particular, weldings in close co-operation with B.V. surveyor Lohmann and his colleague. As far as I remember Lohmann was more active on VIKING 1-5, whilst on VIKING SALLY he was the boss and the other one did the routine work. Anyway Lohmann was always present when major items, such as ramp and visor, were presented by the yard and tested.
My best contacts were Mr. Motikat and Mr. Wahnes. F.B.N representatives came from time to time, however, all paperwork was done by the Sally office in Mariehamn directly with F.B.N. Helsinki. I did some inspections with F.B.N. representatives in Papenburg, e.g.- stairways, lifts;
- boats, liferafts, etc.
- fire doors, fire-fighting equipment, etc.Chief officer Lindström came about 3 months before delivery. I also inspected visor/ramp etc. I checked the visor during building together with the 1st engineer. It was very hectic during the last days before launching because there were delays in delivery of equipment from Sweden due to a strike in this country. After launching the visor was lifted ashore again. It was before only welded to the vessel by steel-bars. At present I do not remember details of the bearing/bushing system of visor hinges.
The bow ramp was considered to be the collision bulkhead same as on all the predecessors built by Meyer Werft, but also the same as on e.g. TURELLA, ROSELLA, VIKING SONG, VIKING SAGA, and many other ferries.
On car deck there were 4 cameras installed, i.e. 1 forward to the forward ramp, 2 at the sides and 1 looking aft to the stern ramp. The cameras could be moved and had also zoom ability. Monitors were on the bridge (located at port side of the entrance to the chart room) and in the engine control room (ECR) above the instrument panel. The camera picture shown on the monitor could be changed to another camera either manually or automatically. The monitors on bridge and in ECR were showing the same picture. The main monitor with operating sticks was in the ECR, i.e. the bridge could not change to another camera.«The complete statement of Lars Karlsson can be found in Enclosure <IP_ADDRESS>.
The vessel was finally taken over by owners on the 29 June 1980 at 23.30 hours at Emden according to the Delivery Certificate attached as Enclosure <IP_ADDRESS> and the "Handing Over Report" attached as Enclosure <IP_ADDRESS>.


cart
food type :{{item.Type}}
farm's name : {{item.Owner}}
costs {{item.cost}}
total cost = {{total}}


Manage your education and skills funding prototype Component library
Components are reusable parts of the user interface that have been used in the design of the MYESF service, sub services and features.
All of the listed components are made up from the GOV.UK Design System styles, patterns and components. It is also based on the GOV.UK elements design components.


---
layout: post
title: no title
date: '2016-09-13T00:40:07-04:00'
tags:
- processing
- p5
- progress
tumblr_url: https://alexcodes.tumblr.com/post/150342964194/an-iteration-of-the-previous-sketch-working-on
---
An iteration of the previous sketch. Working on converting it to P5 to make it interactive.


W e l c o m e
Total score :
Current frame :
Current roll :
Bowl
New Game
Roll 1 :
Roll 2 :
You have
more pins to hit


truste
Businessmodel of TRUSTe
Customer Segments
TRUSTe primarily serves business customers, notably those that manage an online user base and as such are required to comply with privacy regulations. This principally comprises websites – media, retail and business – mobile application developers, online advertisers, and companies that maintain an online software platform.
Among its customers, TRUSTe names media group The New York Times; retailers GAP, McDonalds and Nestle; tech companies Apple, Oracle and PayPal; and app developers Zynga and App Annie.
Value Propositions
TRUSTe is an established and experienced name in the privacy certification space, with a brand reputation for reliability and quality. It is thus an attractive provider for new customers.
The Company provides a full suite of products and services that allow customers to manage their privacy needs across multiple channels – including websites, applications, advertising and cloud platforms – from one single provider, making the process more efficient and simple for customers.
TRUSTe also provides its client ongoing and personal care to its customers, with flexible support afforded to clients of all sizes and across all sectors.
Channels
TRUSTe’s products and services can be acquired through direct consultation with the Company’s sales and marketing team. Ongoing management of user accounts and subscriptions can be accessed through the Company’s website at www.truste.com, as well as by using a variety of help pages and tutorials.
TRUSTe also allows users to access the TRUSTe Privacy app – available through iOS and Google Play as a free download – which allows customers to manage its advertising privacy needs.
Customer Relationships
TRUSTe provides a tailored service to its customers. Before purchasing products and services from the Company, potential clients must have a consultation with members of TRUSTe’s sales and marketing team.
Users of the TRUSTe Privacy Management Platform also receive ongoing care from the Company. User accounts can be accessed in a self-service form, and many help and tutorials are available on-hand via the Company’s website, including advice regarding privacy disputes.
Key Activities
TRUSTe provides privacy monitoring controls for customers with an online user base and privacy assessment automation products that enable users to the continues compliance of customers.
The Company also provides privacy consulting services and certifications and assessments, with compliant websites, applications and platforms being awarded a TRUSTe verification certificate that confirms it complies with all relevant privacy regulations.
Key Partners
TRUSTe has a number of industry partners comprising regulatory and compliance bodies, human rights non-profits and trade associations. The Company’s partners are listed on its website and include the Future of Privacy Forum, the Digital Advertising Alliance, the Centre for Democracy and Technology, the Internet Advertising Bureau and the National Cyber Security Alliance.
TRUSTe also partners with law firms Baker and Mckenzie, Alston and Bird, Cooley, and Davis Wright Tremaine in order to ensure that they remain compliant with all current privacy laws. To this end the Company also entered a partnership with contract management provider PactSafe in 2015.
Key Resources
TRUSTe’s key resources are its proprietary technology and software and its personnel. The Company’s privacy tools, upon which its credibility as a privacy compliance assessor are based, are the Company’s most important asset.
Searches of records held by the US Patent and Trademark Office, however, identified no patent applications registered in the Company’s name.
TRUSTe’s sales and marketing team, which is responsible for on-boarding new clients and offering support to existing customers, is a key component of the Company’s operations.
TRUSTe also relies on the reliable maintenance of its online platform and the infrastructure including server space, required to operate it.
Cost Structure
TRUSTe incurs costs related to the maintenance and development of its technology platform and infrastructure and the retention of its personnel, notably its large sales and marketing team.
The Company also accrues fixed costs through the operation of its offices in San Francisco, California and London, England.
Revenue Streams
TRUSTe generates revenue through the sale of various products and services related to online privacy compliance. This includes monthly and annual subscriptions to the TRUSTE Data Privacy Management Platform, which includes a variety of privacy tools, as well one-off payments for assessments and certifications. Subscription services begin at around $1,000 per month, with Corporate and Enterprise memberships costing more.
TRUSTe also generates revenue through paid-for consultation services, including the running of privacy workshops of risk assessments.


# tuning notebook template for jupyter
from IPython.core.display import display, HTML
display(HTML("<style>.container { width:100% !important; }</style>"))
In this project, you'll use generative adversarial networks to generate new images of faces.
You'll be using two datasets in this project:
Since the celebA dataset is complex and you're doing GANs in a project for the first time, we want you to test your neural network on MNIST before CelebA. Running the GANs on MNIST will allow you to see how well your model trains sooner.
If you're using FloydHub, set data_dir
to "/input" and use the FloydHub data ID "R5KrjnANiKVhLWAkpXhNBe".
# DEV LOCAL: Approach to download the input dataset to dev on my local computer
# the TS examples does not have MIST, which I need to download manually and store under ./data/img_align_celeba/
# from tensorflow.examples.tutorials.mnist import input_data
# mnist_images = input_data.read_data_sets('MNIST_data')
# data_dir = './data'
# data_dir = './MNIST_data'
# FloydHub - Use with data ID "R5KrjnANiKVhLWAkpXhNBe"
data_dir = '/input'
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import helper
helper.download_extract('mnist', data_dir)
helper.download_extract('celeba', data_dir)
import helper
show_n_images = 25
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
%matplotlib inline
import os
from glob import glob
from matplotlib import pyplot
mnist_images = helper.get_batch(glob(os.path.join(data_dir, 'mnist/*.jpg'))[:show_n_images], 28, 28, 'L')
pyplot.imshow(helper.images_square_grid(mnist_images, 'L'), cmap='gray')
The CelebFaces Attributes Dataset (CelebA) dataset contains over 200,000 celebrity images with annotations. Since you're going to be generating faces, you won't need the annotations. You can view the first number of examples by changing show_n_images
.
show_n_images = 25
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
mnist_images = helper.get_batch(glob(os.path.join(data_dir, 'img_align_celeba/*.jpg'))[:show_n_images], 28, 28, 'RGB')
pyplot.imshow(helper.images_square_grid(mnist_images, 'RGB'))
Since the project's main focus is on building the GANs, we'll preprocess the data for you. The values of the MNIST and CelebA dataset will be in the range of -0.5 to 0.5 of 28x28 dimensional images. The CelebA images will be cropped to remove parts of the image that don't include a face, then resized down to 28x28.

You'll build the components necessary to build a GANs by implementing the following functions below:
model_inputs
discriminator
generator
model_loss
model_opt
train
This will check to make sure you have the correct version of TensorFlow and access to a GPU
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
from distutils.version import LooseVersion
import warnings
import tensorflow as tf
# Check TensorFlow Version
assert LooseVersion(tf.__version__) >= LooseVersion('1.0'), 'Please use TensorFlow version 1.0 or newer. You are using {}'.format(tf.__version__)
print('TensorFlow Version: {}'.format(tf.__version__))
# Check for a GPU
if not tf.test.gpu_device_name():
warnings.warn('No GPU found. Please use a GPU to train your neural network.')
else:
print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))
Implement the model_inputs
function to create TF Placeholders for the Neural Network. It should create the following placeholders:
image_width
, image_height
, and image_channels
.z_dim
.Return the placeholders in the following the tuple (tensor of real input images, tensor of z data)
import problem_unittests as tests
def model_inputs(image_width, image_height, image_channels, z_dim):
"""
Create the model inputs
:param image_width: The input image width
:param image_height: The input image height
:param image_channels: The number of image channels
:param z_dim: The dimension of Z
:return: Tuple of (tensor of real input images, tensor of z data, learning rate)
"""
# DONE: Implement Function
real_dim = (image_width, image_height, image_channels)
inputs_real = tf.placeholder(tf.float32, (None, *real_dim), name='input_real')
inputs_z = tf.placeholder(tf.float32, (None, z_dim), name='input_z')
learning_rate = tf.placeholder(tf.float32, None, name='learning_rate')
return inputs_real, inputs_z, learning_rate
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_model_inputs(model_inputs)
Implement discriminator
to create a discriminator neural network that discriminates on images
. This function should be able to reuse the variables in the neural network. Use tf.variable_scope
with a scope name of "discriminator" to allow the variables to be reused. The function should return a tuple of (tensor output of the discriminator, tensor logits of the discriminator).
def discriminator(images, reuse=False):
"""
Create the discriminator network
:param images: Tensor of input image(s)
:param reuse: Boolean if the weights should be reused
:return: Tuple of (tensor output of the discriminator, tensor logits of the discriminator)
"""
# DONE: Implement Function
alpha = 0.1
with tf.variable_scope('discriminator', reuse=reuse):
layer_1 = tf.layers.conv2d(images, 128, 5, strides=2, padding='same',
kernel_initializer=tf.contrib.layers.xavier_initializer())
relu_1 = tf.maximum(alpha * layer_1, layer_1)
# 14x14x128; 5
layer_2 = tf.layers.conv2d(relu_1, 256, 5, strides=2, padding='same',
kernel_initializer=tf.contrib.layers.xavier_initializer())
bn_2 = tf.layers.batch_normalization(layer_2, training=True)
relu_2 = tf.maximum(alpha * bn_2, bn_2)
# 7x7x256; 5
layer_3 = tf.layers.conv2d(relu_2, 512, 5, strides=2, padding='same',
kernel_initializer=tf.contrib.layers.xavier_initializer())
bn_3 = tf.layers.batch_normalization(layer_3, training=True)
relu_3 = tf.maximum(alpha * bn_3, bn_3)
# 4x4x512; 5
layer_4 = tf.layers.conv2d(relu_3, 512, 2, strides=2, padding='same',
kernel_initializer=tf.contrib.layers.xavier_initializer())
bn_4 = tf.layers.batch_normalization(layer_4, training=True)
relu_4 = tf.maximum(alpha * bn_4, bn_4)
# 4x4x512
flattened = tf.reshape(relu_4, (-1, 4*4*512))
# 1x8192
# tensor output of the discriminator
logits = tf.layers.dense(flattened, 1)
# tensor logits of the discriminator)
output = tf.sigmoid(logits)
return output, logits
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_discriminator(discriminator, tf)
Implement generator
to generate an image using z
. This function should be able to reuse the variables in the neural network. Use tf.variable_scope
with a scope name of "generator" to allow the variables to be reused. The function should return the generated 28 x 28 x out_channel_dim
images.
def generator(z, out_channel_dim, is_train=True):
"""
Create the generator network
:param z: Input z
:param out_channel_dim: The number of channels in the output image
:param is_train: Boolean if generator is being used for training
:return: The tensor output of the generator
"""
# DONE: Implement Function
alpha = 0.1
with tf.variable_scope('generator', reuse=not is_train):
layer_1 = tf.layers.dense(z, 7*7*512)
layer_1 = tf.reshape(layer_1, (-1, 7, 7, 512))
bn_1 = tf.layers.batch_normalization(layer_1, training=is_train)
relu_1 = tf.maximum(alpha * bn_1, bn_1)
# 7x7x512
layer_2 = tf.layers.conv2d_transpose(relu_1, 256, 2, strides=2, padding='same',
kernel_initializer=tf.contrib.layers.xavier_initializer())
bn_2 = tf.layers.batch_normalization(layer_2, training=is_train)
relu_2 = tf.maximum(alpha * bn_2, bn_2)
# 14x14x256
layer_3 = tf.layers.conv2d_transpose(relu_2, 128, 3, strides=2, padding='same',
kernel_initializer=tf.contrib.layers.xavier_initializer())
bn_3 = tf.layers.batch_normalization(layer_3, training=is_train)
relu_3 = tf.maximum(alpha * bn_3, bn_3)
# 28x28x128
layer_4 = tf.layers.conv2d_transpose(relu_3, 64, 4, strides=1, padding='same',
kernel_initializer=tf.contrib.layers.xavier_initializer())
bn_4 = tf.layers.batch_normalization(layer_4, training=is_train)
relu_4 = tf.maximum(alpha * bn_4, bn_4)
logits = tf.layers.conv2d_transpose(relu_4, out_channel_dim, 4, strides=1, padding='same',
kernel_initializer=tf.contrib.layers.xavier_initializer())
# Tensor output
out = tf.tanh(logits)
return out
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_generator(generator, tf)
Implement model_loss
to build the GANs for training and calculate the loss. The function should return a tuple of (discriminator loss, generator loss). Use the following functions you implemented:
discriminator(images, reuse=False)
generator(z, out_channel_dim, is_train=True)
def model_loss(input_real, input_z, out_channel_dim):
"""
Get the loss for the discriminator and generator
:param input_real: Images from the real dataset
:param input_z: Z input
:param out_channel_dim: The number of channels in the output image
:return: A tuple of (discriminator loss, generator loss)
"""
# DONE: Implement Function
g_model = generator(input_z, out_channel_dim)
d_model_real, d_logits_real = discriminator(input_real)
d_model_fake, d_logits_fake = discriminator(g_model, reuse=True)
d_loss_real = tf.reduce_mean(
tf.nn.sigmoid_cross_entropy_with_logits(logits=d_logits_real, labels=tf.ones_like(d_model_real)))
d_loss_fake = tf.reduce_mean(
tf.nn.sigmoid_cross_entropy_with_logits(logits=d_logits_fake, labels=tf.zeros_like(d_model_fake)))
g_loss = tf.reduce_mean(
tf.nn.sigmoid_cross_entropy_with_logits(logits=d_logits_fake, labels=tf.ones_like(d_model_fake)))
d_loss = d_loss_real + d_loss_fake
return d_loss, g_loss
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_model_loss(model_loss)
Implement model_opt
to create the optimization operations for the GANs. Use tf.trainable_variables
to get all the trainable variables. Filter the variables with names that are in the discriminator and generator scope names. The function should return a tuple of (discriminator training operation, generator training operation).
def model_opt(d_loss, g_loss, learning_rate, beta1):
"""
Get optimization operations
:param d_loss: Discriminator loss Tensor
:param g_loss: Generator loss Tensor
:param learning_rate: Learning Rate Placeholder
:param beta1: The exponential decay rate for the 1st moment in the optimizer
:return: A tuple of (discriminator training operation, generator training operation)
"""
# DONE: Implement Function
t_vars = tf.trainable_variables()
d_vars = [var for var in t_vars if var.name.startswith('discriminator')]
g_vars = [var for var in t_vars if var.name.startswith('generator')]
update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.control_dependencies(update_ops):
d_train_opt = tf.train.AdamOptimizer(learning_rate, beta1=beta1).minimize(d_loss, var_list=d_vars)
g_train_opt = tf.train.AdamOptimizer(learning_rate, beta1=beta1).minimize(g_loss, var_list=g_vars)
return d_train_opt, g_train_opt
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_model_opt(model_opt, tf)
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import numpy as np
def show_generator_output(sess, n_images, input_z, out_channel_dim, image_mode):
"""
Show example output for the generator
:param sess: TensorFlow session
:param n_images: Number of Images to display
:param input_z: Input Z Tensor
:param out_channel_dim: The number of channels in the output image
:param image_mode: The mode to use for images ("RGB" or "L")
"""
cmap = None if image_mode == 'RGB' else 'gray'
z_dim = input_z.get_shape().as_list()[-1]
example_z = np.random.uniform(-1, 1, size=[n_images, z_dim])
samples = sess.run(
generator(input_z, out_channel_dim, False),
feed_dict={input_z: example_z})
images_grid = helper.images_square_grid(samples, image_mode)
pyplot.imshow(images_grid, cmap=cmap)
pyplot.show()
Implement train
to build and train the GANs. Use the following functions you implemented:
model_inputs(image_width, image_height, image_channels, z_dim)
model_loss(input_real, input_z, out_channel_dim)
model_opt(d_loss, g_loss, learning_rate, beta1)
Use the show_generator_output
to show generator
output while you train. Running show_generator_output
for every batch will drastically increase training time and increase the size of the notebook. It's recommended to print the generator
output every 100 batches.
def train(epoch_count, batch_size, z_dim, learning_rate, beta1, get_batches, data_shape, data_image_mode):
"""
Train the GAN
:param epoch_count: Number of epochs
:param batch_size: Batch Size
:param z_dim: Z dimension
:param learning_rate: Learning Rate
:param beta1: The exponential decay rate for the 1st moment in the optimizer
:param get_batches: Function to get batches
:param data_shape: Shape of the data
:param data_image_mode: The image mode to use for images ("RGB" or "L")
"""
# DONE: Build Modeltrain
# DEV: printing call arameters
print("Parameters are epoch_count(%s), batch_size(%s), z_dim(%s), learning_rate(%s), beta1(%s), data_shape(%s), data_image_mode(%s)"
% (epoch_count, batch_size, z_dim, learning_rate, beta1, data_shape, data_image_mode))
# loging settings
print_step_module = 25
print_sample_module = 100
# Model inputs
# print(data_shape[1])
input_real, input_z, lr = model_inputs(data_shape[1], data_shape[2], data_shape[3], z_dim)
# Model losses
d_loss, g_loss = model_loss(input_real, input_z, data_shape[3])
# Model optimizers
d_opt, g_opt = model_opt(d_loss, g_loss, learning_rate, beta1)
steps = 0
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
for epoch_i in range(epoch_count):
for batch_images in get_batches(batch_size):
# DONE: Train Model
steps += 1
batch_images = batch_images * 2
# Generator random noise
batch_z = np.random.uniform(-1, 1, size=(batch_size, z_dim))
# Run optimizers
_ = sess.run(d_opt, feed_dict={input_real: batch_images, input_z: batch_z,
lr: learning_rate})
_ = sess.run(g_opt, feed_dict={input_real: batch_images, input_z: batch_z,
lr: learning_rate})
if steps % print_step_module == 0:
train_loss_d = d_loss.eval({input_z: batch_z, input_real: batch_images})
train_loss_g = g_loss.eval({input_z: batch_z})
print("Epoch {}: Step {}...".format(epoch_i+1, steps+1),
"Discriminator Loss: {:.4f}...".format(train_loss_d),
"Generator Loss: {:.4f}".format(train_loss_g))
if steps % print_sample_module == 0:
show_generator_output(sess, 36, input_z, data_shape[3], data_image_mode)
Test your GANs architecture on MNIST. After 2 epochs, the GANs should be able to generate images that look like handwritten digits. Make sure the loss of the generator is lower than the loss of the discriminator or close to 0.
batch_size = 32
z_dim = 100
learning_rate = 0.0003
beta1 = 0.5
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
epochs = 2
mnist_dataset = helper.Dataset('mnist', glob(os.path.join(data_dir, 'mnist/*.jpg')))
with tf.Graph().as_default():
train(epochs, batch_size, z_dim, learning_rate, beta1, mnist_dataset.get_batches,
mnist_dataset.shape, mnist_dataset.image_mode)
Run your GANs on CelebA. It will take around 20 minutes on the average GPU to run one epoch. You can run the whole epoch or stop when it starts to generate realistic faces.
batch_size = 64
z_dim = 100
learning_rate = 0.0003
beta1 = 0.5
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
epochs = 4
celeba_dataset = helper.Dataset('celeba', glob(os.path.join(data_dir, 'img_align_celeba/*.jpg')))
with tf.Graph().as_default():
train(epochs, batch_size, z_dim, learning_rate, beta1, celeba_dataset.get_batches,
celeba_dataset.shape, celeba_dataset.image_mode)
When submitting this project, make sure to run all the cells before saving the notebook. Save the notebook file as "dlnd_face_generation.ipynb" and save it as a HTML file under "File" -> "Download as". Include the "helper.py" and "problem_unittests.py" files in your submission.


Click the symbol on the map to find the nearest dog poop bag dispenser!
.
Click me to report a missing dispenser
I don't do anything yet


Extracts a glimpse from the input tensor.
tf.compat.v2.image.extract_glimpse
tf.image.extract_glimpse( input, size, offsets, centered=True, normalized=True, noise='uniform', name=None )
Returns a set of windows called glimpses extracted at location
offsets
from the input tensor. If the windows only partially
overlaps the inputs, the non overlapping areas will be filled with
random noise.
The result is a 4-D tensor of shape [batch_size, glimpse_height,
glimpse_width, channels]
. The channels and batch dimensions are the
same as that of the input tensor. The height and width of the output
windows are specified in the size
parameter.
The argument normalized
and centered
controls how the windows are built:
input
: A Tensor
of type float32
. A 4-D float tensor of shape
[batch_size, height, width, channels]
.size
: A Tensor
of type int32
. A 1-D tensor of 2 elements containing the
size of the glimpses to extract. The glimpse height must be specified
first, following by the glimpse width.offsets
: A Tensor
of type float32
. A 2-D integer tensor of shape
[batch_size, 2]
containing the y, x locations of the center of each
window.centered
: An optional bool
. Defaults to True
. indicates if the offset
coordinates are centered relative to the image, in which case the (0, 0)
offset is relative to the center of the input images. If false, the (0,0)
offset corresponds to the upper left corner of the input images.normalized
: An optional bool
. Defaults to True
. indicates if the offset
coordinates are normalized.noise
: An optional string
. Defaults to uniform
. indicates if the noise
should be uniform
(uniform distribution), gaussian
(gaussian
distribution), or zero
(zero padding).name
: A name for the operation (optional).A Tensor
of type float32
.
python
BATCH_SIZE = 1
IMAGE_HEIGHT = 3
IMAGE_WIDTH = 3
CHANNELS = 1
GLIMPSE_SIZE = (2, 2)
image = tf.reshape(tf.range(9, delta=1, dtype=tf.float32),
shape=(BATCH_SIZE, IMAGE_HEIGHT, IMAGE_WIDTH, CHANNELS))
output = tf.image.extract_glimpse(image, size=GLIMPSE_SIZE,
offsets=[[1, 1]], centered=False, normalized=False)


{% load static %} {% block content %}
Please note that this application is currently in beta and is still undergoing final testing before its official release.
Disclaimer: The information provided by the Texas Water Development Board and the Texas Natural Resources Information System (TNRIS) in the Lakes of Texas web application is for general information purposes only. All information on the site/application is provided in good faith, however we make no representation or warranty of any kind, express or implied, regarding the accuracy, adequacy, validity, reliability, availability or completeness of any information on the site/application.


Thousands in Philippines flee Typhoon Ramon
Heavy rains are forecast to swell rivers, cut off roads, and threaten low-lying communities in Cagayan province, where officials guided residents towards shelter
by Agence France-PresseSubscribe
Email Address
Newsletters
- Glenda Gloria's Take this Tuesday
- Huddle by Miriam Grace Go
- The Newsbreak Agenda by Chay Hofileña
- Sometimes, all you need is video by Lilibeth Frondoso


The Voyages of Doctor Dolittle
by Hugh Lofting
Prologue
Additional Information
- Year Published: 1922
- Language: English
- Country of Origin: England
- Source: Lofting, H. (1922). The Voyages of Doctor Dolittle. London, England: Lippincott Publishing.
-
Readability:
- Flesch–Kincaid Level: 4.2
- Word Count: 272
- Genre: Fantasy
- Keywords: 20th century literature, british literature, children's literature
- ✎ Cite This
Downloads
All that I have written so far about Doctor Dolittle I heard long after it happened from those who had known him—indeed a great deal of it took place before I was born. But I now come to set down that part of the great man's life which I myself saw and took part in.
Many years ago the Doctor gave me permission to do this. But we were both of us so busy then voyaging around the world, having adventures and filling note-books full of natural history that I never seemed to get time to sit down and write of our doings.
Now of course, when I am quite an old man, my memory isn't so good any more. But whenever I am in doubt and have to hesitate and think, I always ask Polynesia, the parrot.
That wonderful bird (she is now nearly two hundred and fifty years old) sits on the top of my desk, usually humming sailor songs to herself, while I write this book. And, as every one who ever met her knows, Polynesia's memory is the most marvelous memory in the world. If there is any happening I am not quite sure of, she is always able to put me right, to tell me exactly how it took place, who was there and everything about it. In fact sometimes I almost think I ought to say that this book was written by Polynesia instead of me.
Very well then, I will begin. And first of all I must tell you something about myself and how I came to meet the Doctor.


Service schedule group Details
Service schedule group id
Service schedule group label
Calendars
Calendar ID
Service label
Start Date
End Date
Sun
Mon
Tue
Wed
Thu
Fri
Sat
Action


| PERL5263DELTA(1) | Perl Programmers Reference Guide | PERL5263DELTA(1) |
If you are upgrading from an earlier release such as 5.26.1, first read perl5262delta, which describes differences between 5.26.1 and 5.26.2.
[perl #133250] <https://rt.perl.org/Ticket/Display.html?id=133250> [cpan #125523] <https://rt.cpan.org/Ticket/Display.html?id=125523>
[perl #133204] <https://rt.perl.org/Ticket/Display.html?id=133204>
[perl #133423] <https://rt.perl.org/Ticket/Display.html?id=133423>
[perl #133192] <https://rt.perl.org/Ticket/Display.html?id=133192>
[perl #131649] <https://rt.perl.org/Ticket/Display.html?id=131649>
(F) While parsing an extended character class a ']' character was encountered at a point in the definition where the only legal use of ']' is to close the character class definition as part of a '])', you may have forgotten the close paren, or otherwise confused the parser.
(F) While parsing a nested extended character class like:
(?[ ... (?flags:(?[ ... ])) ... ]) ^
we expected to see a close paren ')' (marked by ^) but did not.
(F) While parsing a nested extended character class like:
(?[ ... (?flags:(?[ ... ])) ... ]) ^
we expected to see a close paren ')' (marked by ^) but did not.
This fatal error message has been slightly expanded (from "Syntax error in (?[...]) in regex m/%s/") for greater clarity.
Excluding auto-generated files, documentation and release tools, there were approximately 770 lines of changes to 10 .pm, .t, .c and .h files.
Perl continues to flourish into its third decade thanks to a vibrant community of users and developers. The following people are known to have contributed the improvements that became Perl 5.26.3:
Aaron Crane, Abigail, Chris 'BinGOs' Williams, Dagfinn Ilmari Mannsaaker, David Mitchell, H.Merijn Brand, James E Keenan, John SJ Anderson, Karen Etheridge, Karl Williamson, Sawyer X, Steve Hay, Todd Rinaldo, Tony Cook, Yves Orton.
The list above is almost certainly incomplete as it is automatically generated from version control history. In particular, it does not include the names of the (very much appreciated) contributors who reported issues to the Perl bug tracker.
Many of the changes included in this version originated in the CPAN modules included in Perl's core. We're grateful to the entire CPAN community for helping Perl to flourish.
For a more complete list of all of Perl's historical contributors, please see the AUTHORS file in the Perl source distribution.
If you believe you have an unreported bug, please run the perlbug program included with your release. Be sure to trim your bug down to a tiny but sufficient test case. Your bug report, along with the output of "perl -V", will be sent off to<EMAIL_ADDRESS>to be analysed by the Perl porting team.
If the bug you are reporting has security implications which make it inappropriate to send to a publicly archived mailing list, then see "SECURITY VULNERABILITY CONTACT INFORMATION" in perlsec for details of how to report the issue.
perlthanks
This will send an email to the Perl 5 Porters list with your show of thanks.
The INSTALL file for how to build Perl.
The README file for general stuff.
The Artistic and Copying files for copyright information.
| 2019-10-21 | perl v5.30.3 |


Store photos / articles
Organize photos / articles
Browse member collections
Save to read later
Whether it's Biology, Physics, Chemistry or Mathematics or ...?
Create, organize, store or share your photos to help you study or
collaborate efficiently.
Store photos / articles
Organize photos / articles
Browse member collections
Save to read later


Processors
All processors are defined in the following way within a pipeline definition:
{
"PROCESSOR_NAME" : {
... processor configuration options ...
}
}
Each processor defines its own configuration parameters, but all processors have
the ability to declare tag
, on_failure
and if
fields. These fields are optional.
A tag
is simply a string identifier of the specific instantiation of a certain
processor in a pipeline. The tag
field does not affect the processor’s behavior,
but is very useful for bookkeeping and tracing errors to specific processors.
The if
field must contain a script that returns a boolean value. If the script evaluates to true
then the processor will be executed for the given document otherwise it will be skipped.
The if
field takes an object with the script fields defined in script-options
and accesses a read only version of the document via the same ctx
variable used by scripts in the
Script processor.
{
"set": {
"if": "ctx.foo == 'someValue'",
"field": "found",
"value": true
}
}
See Conditional Execution in Pipelines to learn more about the if
field and conditional execution.
See Handling Failures in Pipelines to learn more about the on_failure
field and error handling in pipelines.
The node info API will provide a per node list of what processors are available.
Custom processors must be installed on all nodes. The put pipeline API will fail if a processor specified in a pipeline
doesn’t exist on all nodes. If you rely on custom processor plugins make sure to mark these plugins as mandatory by adding
plugin.mandatory
setting to the config/opensearch.yml
file, for example:
plugin.mandatory: ingest-attachment
A node will not start if this plugin is not available.
The node stats API can be used to fetch ingest usage statistics, globally and on a per pipeline basis. Useful to find out which pipelines are used the most or spent the most time on preprocessing.
Ingest Processor Plugins
Additional ingest processors can be implemented and installed as OpenSearch plugins. See Ingest plugins for information about the available ingest plugins.
Append processor
Appends one or more values to an existing array if the field already exists and it is an array. Converts a scalar to an array and appends one or more values to it if the field exists and it is a scalar. Creates an array containing the provided values if the field doesn’t exist. Accepts a single value or an array of values.
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
The field to be appended to. Supports template snippets. |
|
yes |
- |
The value to be appended. Supports template snippets. |
|
no |
true |
If |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
{
"append": {
"field": "tags",
"value": ["production", "{{app}}", "{{owner}}"]
}
}
Bytes processor
Converts a human readable byte value (e.g. 1kb) to its value in bytes (e.g. 1024). If the field is an array of strings, all members of the array will be converted.
Supported human readable units are "b", "kb", "mb", "gb", "tb", "pb" case insensitive. An error will occur if the field is not a supported format or resultant value exceeds 2^63.
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
The field to convert |
|
no |
|
The field to assign the converted value to, by default |
|
no |
|
If |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
{
"bytes": {
"field": "file.size"
}
}
Convert processor
Converts a field in the currently ingested document to a different type, such as converting a string to an integer. If the field value is an array, all members will be converted.
The supported types include: integer
, long
, float
, double
, string
, boolean
, and auto
.
Specifying boolean
will set the field to true if its string value is equal to true
(ignore case), to
false if its string value is equal to false
(ignore case), or it will throw an exception otherwise.
Specifying auto
will attempt to convert the string-valued field
into the closest non-string type.
For example, a field whose value is "true"
will be converted to its respective boolean type: true
. Do note
that float takes precedence of double in auto
. A value of "242.15"
will "automatically" be converted to
242.15
of type float
. If a provided field cannot be appropriately converted, the Convert Processor will
still process successfully and leave the field value as-is. In such a case, target_field
will
still be updated with the unconverted field value.
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
The field whose value is to be converted |
|
no |
|
The field to assign the converted value to, by default |
|
yes |
- |
The type to convert the existing value to |
|
no |
|
If |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
PUT _ingest/pipeline/my-pipeline-id
{
"description": "converts the content of the id field to an integer",
"processors" : [
{
"convert" : {
"field" : "id",
"type": "integer"
}
}
]
}
CSV processor
Extracts fields from CSV line out of a single text field within a document. Any empty field in CSV will be skipped.
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
The field to extract data from |
|
yes |
- |
The array of fields to assign extracted values to |
|
no |
, |
Separator used in CSV, has to be single character string |
|
no |
" |
Quote used in CSV, has to be single character string |
|
no |
|
If |
|
no |
|
Trim whitespaces in unquoted fields |
|
no |
- |
Value used to fill empty fields, empty fields will be skipped if this is not provided.
Empty field is one with no value (2 consecutive separators) or empty quotes ( |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
{
"csv": {
"field": "my_field",
"target_fields": ["field1", "field2"]
}
}
If the trim
option is enabled then any whitespace in the beginning and in the end of each unquoted field will be trimmed.
For example with configuration above, a value of A, B
will result in field field2
having value B
(with space at the beginning). If trim
is enabled A, B
will result in field field2
having value B
(no whitespace). Quoted fields will be left untouched.
Date processor
Parses dates from fields, and then uses the date or timestamp as the timestamp for the document.
By default, the date processor adds the parsed date as a new field called @timestamp
. You can specify a
different field by setting the target_field
configuration parameter. Multiple date formats are supported
as part of the same date processor definition. They will be used sequentially to attempt parsing the date field,
in the same order they were defined as part of the processor definition.
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
The field to get the date from. |
|
no |
@timestamp |
The field that will hold the parsed date. |
|
yes |
- |
An array of the expected date formats. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N. |
|
no |
UTC |
The timezone to use when parsing the date. Supports template snippets. |
|
no |
ENGLISH |
The locale to use when parsing the date, relevant when parsing month names or week days. Supports template snippets. |
|
no |
|
The format to use when writing the date to |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
Here is an example that adds the parsed date to the timestamp
field based on the initial_date
field:
{
"description" : "...",
"processors" : [
{
"date" : {
"field" : "initial_date",
"target_field" : "timestamp",
"formats" : ["dd/MM/yyyy HH:mm:ss"],
"timezone" : "Europe/Amsterdam"
}
}
]
}
The timezone
and locale
processor parameters are templated. This means that their values can be
extracted from fields within documents. The example below shows how to extract the locale/timezone
details from existing fields, my_timezone
and my_locale
, in the ingested document that contain
the timezone and locale values.
{
"description" : "...",
"processors" : [
{
"date" : {
"field" : "initial_date",
"target_field" : "timestamp",
"formats" : ["ISO8601"],
"timezone" : "{{my_timezone}}",
"locale" : "{{my_locale}}"
}
}
]
}
Date index name processor
The purpose of this processor is to point documents to the right time based index based on a date or timestamp field in a document by using the date math index name support.
The processor sets the _index
metadata field with a date math index name expression based on the provided index name
prefix, a date or timestamp field in the documents being processed and the provided date rounding.
First, this processor fetches the date or timestamp from a field in the document being processed. Optionally, date formatting can be configured on how the field’s value should be parsed into a date. Then this date, the provided index name prefix and the provided date rounding get formatted into a date math index name expression. Also here optionally date formatting can be specified on how the date should be formatted into a date math index name expression.
An example pipeline that points documents to a monthly index that starts with a my-index-
prefix based on a
date in the date1
field:
PUT _ingest/pipeline/monthlyindex
{
"description": "monthly date-time index naming",
"processors" : [
{
"date_index_name" : {
"field" : "date1",
"index_name_prefix" : "my-index-",
"date_rounding" : "M"
}
}
]
}
Using that pipeline for an index request:
PUT /my-index/_doc/1?pipeline=monthlyindex
{
"date1" : "2016-04-25T12:02:01.789Z"
}
{
"_index" : "my-index-2016-04-01",
"_type" : "_doc",
"_id" : "1",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 55,
"_primary_term" : 1
}
The above request will not index this document into the my-index
index, but into the my-index-2016-04-01
index because
it was rounded by month. This is because the date-index-name-processor overrides the _index
property of the document.
To see the date-math value of the index supplied in the actual index request which resulted in the above document being
indexed into my-index-2016-04-01
we can inspect the effects of the processor using a simulate request.
POST _ingest/pipeline/_simulate
{
"pipeline" :
{
"description": "monthly date-time index naming",
"processors" : [
{
"date_index_name" : {
"field" : "date1",
"index_name_prefix" : "my-index-",
"date_rounding" : "M"
}
}
]
},
"docs": [
{
"_source": {
"date1": "2016-04-25T12:02:01.789Z"
}
}
]
}
and the result:
{
"docs" : [
{
"doc" : {
"_id" : "_id",
"_index" : "<my-index-{2016-04-25||/M{yyyy-MM-dd|UTC}}>",
"_type" : "_doc",
"_source" : {
"date1" : "2016-04-25T12:02:01.789Z"
},
"_ingest" : {
"timestamp" : "2016-11-08T19:43:03.850+0000"
}
}
}
]
}
The above example shows that _index
was set to <my-index-{2016-04-25||/M{yyyy-MM-dd|UTC}}>
. OpenSearch
understands this to mean 2016-04-01
as is explained in the date math index name documentation
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
The field to get the date or timestamp from. |
|
no |
- |
A prefix of the index name to be prepended before the printed date. Supports template snippets. |
|
yes |
- |
How to round the date when formatting the date into the index name. Valid values are: |
|
no |
yyyy-MM-dd'T'HH:mm:ss.SSSXX |
An array of the expected date formats for parsing dates / timestamps in the document being preprocessed. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N. |
|
no |
UTC |
The timezone to use when parsing the date and when date math index supports resolves expressions into concrete index names. |
|
no |
ENGLISH |
The locale to use when parsing the date from the document being preprocessed, relevant when parsing month names or week days. |
|
no |
yyyy-MM-dd |
The format to be used when printing the parsed date into the index name. A valid java time pattern is expected here. Supports template snippets. |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
Dissect processor
Similar to the Grok Processor, dissect also extracts structured fields out of a single text field within a document. However unlike the Grok Processor, dissect does not use Regular Expressions. This allows dissect’s syntax to be simple and for some cases faster than the Grok Processor.
Dissect matches a single text field against a defined pattern.
For example the following pattern:
%{clientip} %{ident} %{auth} [%{@timestamp}] \"%{verb} %{request} HTTP/%{httpversion}\" %{status} %{size}
will match a log line of this format:
<IP_ADDRESS> - - [30/Apr/1998:22:00:52 +0000] \"GET /english/venues/cities/images/montpellier/18.gif HTTP/1.0\" 200 3171
and result in a document with the following fields:
"doc": {
"_index": "_index",
"_type": "_type",
"_id": "_id",
"_source": {
"request": "/english/venues/cities/images/montpellier/18.gif",
"auth": "-",
"ident": "-",
"verb": "GET",
"@timestamp": "30/Apr/1998:22:00:52 +0000",
"size": "3171",
"clientip": "<IP_ADDRESS>",
"httpversion": "1.0",
"status": "200"
}
}
A dissect pattern is defined by the parts of the string that will be discarded. In the example above the first part
to be discarded is a single space. Dissect finds this space, then assigns the value of clientip
is everything up
until that space.
Later dissect matches the [
and then ]
and then assigns @timestamp
to everything in-between [
and ]
.
Paying special attention the parts of the string to discard will help build successful dissect patterns.
Successful matches require all keys in a pattern to have a value. If any of the %{keyname}
defined in the pattern do
not have a value, then an exception is thrown and may be handled by the on_failure directive.
An empty key %{}
or a named skip key can be used to match values, but exclude the value from
the final document. All matched values are represented as string data types. The convert processor
may be used to convert to expected data type.
Dissect also supports key modifiers that can change dissect’s default behavior. For example you can instruct dissect to ignore certain fields, append fields, skip over padding, etc. See below for more information.
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
The field to dissect |
|
yes |
- |
The pattern to apply to the field |
|
no |
"" (empty string) |
The character(s) that separate the appended fields. |
|
no |
false |
If |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
{
"dissect": {
"field": "message",
"pattern" : "%{clientip} %{ident} %{auth} [%{@timestamp}] \"%{verb} %{request} HTTP/%{httpversion}\" %{status} %{size}"
}
}
Dissect key modifiers
Key modifiers can change the default behavior for dissection. Key modifiers may be found on the left or right
of the %{keyname}
always inside the %{
and }
. For example %{+keyname →}
has the append and right padding
modifiers.
| Modifier | Name | Position | Example | Description | Details |
|---|---|---|---|---|---|
|
Skip right padding |
(far) right |
|
Skips any repeated characters to the right |
|
|
Append |
left |
|
Appends two or more fields together |
|
|
Append with order |
left and right |
|
Appends two or more fields together in the order specified |
|
|
Named skip key |
left |
|
Skips the matched value in the output. Same behavior as |
|
|
Reference keys |
left |
|
Sets the output key as value of |
Right padding modifier (→
)
The algorithm that performs the dissection is very strict in that it requires all characters in the pattern to match
the source string. For example, the pattern %{fookey} %{barkey}
(1 space), will match the string "foo bar"
(1 space), but will not match the string "foo bar" (2 spaces) since the pattern has only 1 space and the
source string has 2 spaces.
The right padding modifier helps with this case. Adding the right padding modifier to the pattern %{fookey→} %{barkey}
,
It will now will match "foo bar" (1 space) and "foo bar" (2 spaces)
and even "foo bar" (10 spaces).
Use the right padding modifier to allow for repetition of the characters after a %{keyname→}
.
The right padding modifier may be placed on any key with any other modifiers. It should always be the furthest right
modifier. For example: %{+keyname/1→}
and %{→}
Right padding modifier example
Pattern |
|
Input |
1998-08-10T17:15:42,466 WARN |
Result |
* ts = 1998-08-10T17:15:42,466 |
The right padding modifier may be used with an empty key to help skip unwanted data. For example, the same input string, but wrapped with brackets requires the use of an empty right padded key to achieve the same result.
Right padding modifier with empty key example
Pattern |
|
Input |
[1998-08-10T17:15:42,466] [WARN] |
Result |
* ts = 1998-08-10T17:15:42,466 |
Append modifier (+
)
Dissect supports appending two or more results together for the output. Values are appended left to right. An append separator can be specified. In this example the append_separator is defined as a space.
Append modifier example
Pattern |
|
Input |
john jacob jingleheimer schmidt |
Result |
* name = john jacob jingleheimer schmidt |
Append with order modifier (+
and /n
)
Dissect supports appending two or more results together for the output.
Values are appended based on the order defined (/n
). An append separator can be specified.
In this example the append_separator is defined as a comma.
Append with order modifier example
Pattern |
|
Input |
john jacob jingleheimer schmidt |
Result |
* name = schmidt,john,jingleheimer,jacob |
Named skip key (?
)
Dissect supports ignoring matches in the final result. This can be done with an empty key %{}
, but for readability
it may be desired to give that empty key a name.
Named skip key modifier example
Pattern |
|
Input |
<IP_ADDRESS> - - [30/Apr/1998:22:00:52 +0000] |
Result |
* clientip = <IP_ADDRESS> |
Reference keys (*
and &
)
Dissect support using parsed values as the key/value pairings for the structured content. Imagine a system that partially logs in key/value pairs. Reference keys allow you to maintain that key/value relationship.
Reference key modifier example
Pattern |
|
Input |
[2018-08-10T17:15:42,466] [ERR] ip:<IP_ADDRESS> error:REFUSED |
Result |
* ts = 1998-08-10T17:15:42,466 |
Dot expander processor
Expands a field with dots into an object field. This processor allows fields with dots in the name to be accessible by other processors in the pipeline. Otherwise these fields can’t be accessed by any processor.
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
The field to expand into an object field |
|
no |
- |
The field that contains the field to expand. Only required if the field to expand is part another object field, because the |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
{
"dot_expander": {
"field": "foo.bar"
}
}
For example the dot expand processor would turn this document:
{
"foo.bar" : "value"
}
into:
{
"foo" : {
"bar" : "value"
}
}
If there is already a bar
field nested under foo
then
this processor merges the foo.bar
field into it. If the field is
a scalar value then it will turn that field into an array field.
For example, the following document:
{
"foo.bar" : "value2",
"foo" : {
"bar" : "value1"
}
}
is transformed by the dot_expander
processor into:
{
"foo" : {
"bar" : ["value1", "value2"]
}
}
If any field outside of the leaf field conflicts with a pre-existing field of the same name, then that field needs to be renamed first.
Consider the following document:
{
"foo": "value1",
"foo.bar": "value2"
}
Then the foo
needs to be renamed first before the dot_expander
processor is applied. So in order for the foo.bar
field to properly
be expanded into the bar
field under the foo
field the following
pipeline should be used:
{
"processors" : [
{
"rename" : {
"field" : "foo",
"target_field" : "foo.bar""
}
},
{
"dot_expander": {
"field": "foo.bar"
}
}
]
}
The reason for this is that Ingest doesn’t know how to automatically cast a scalar field to an object field.
Drop processor
Drops the document without raising any errors. This is useful to prevent the document from getting indexed based on some condition.
| Name | Required | Default | Description |
|---|---|---|---|
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
{
"drop": {
"if" : "ctx.network_name == 'Guest'"
}
}
Fail processor
Raises an exception. This is useful for when you expect a pipeline to fail and want to relay a specific message to the requester.
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
The error message thrown by the processor. Supports template snippets. |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
{
"fail": {
"if" : "ctx.tags.contains('production') != true",
"message": "The production tag is not present, found tags: {{tags}}"
}
}
Foreach processor
Processes elements in an array of unknown length.
All processors can operate on elements inside an array, but if all elements of an array need to
be processed in the same way, defining a processor for each element becomes cumbersome and tricky
because it is likely that the number of elements in an array is unknown. For this reason the foreach
processor exists. By specifying the field holding array elements and a processor that
defines what should happen to each element, array fields can easily be preprocessed.
A processor inside the foreach processor works in the array element context and puts that in the ingest metadata
under the _ingest._value
key. If the array element is a json object it holds all immediate fields of that json object.
and if the nested object is a value is _ingest._value
just holds that value. Note that if a processor prior to the
foreach
processor used _ingest._value
key then the specified value will not be available to the processor inside
the foreach
processor. The foreach
processor does restore the original value, so that value is available to processors
after the foreach
processor.
Note that any other field from the document are accessible and modifiable like with all other processors. This processor
just puts the current array element being read into _ingest._value
ingest metadata attribute, so that it may be
pre-processed.
If the foreach
processor fails to process an element inside the array, and no on_failure
processor has been specified,
then it aborts the execution and leaves the array unmodified.
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
The array field |
|
yes |
- |
The processor to execute against each field |
|
no |
false |
If |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
Assume the following document:
{
"values" : ["foo", "bar", "baz"]
}
When this foreach
processor operates on this sample document:
{
"foreach" : {
"field" : "values",
"processor" : {
"uppercase" : {
"field" : "_ingest._value"
}
}
}
}
Then the document will look like this after preprocessing:
{
"values" : ["FOO", "BAR", "BAZ"]
}
Let’s take a look at another example:
{
"persons" : [
{
"id" : "1",
"name" : "John Doe"
},
{
"id" : "2",
"name" : "Jane Doe"
}
]
}
In this case, the id
field needs to be removed,
so the following foreach
processor is used:
{
"foreach" : {
"field" : "persons",
"processor" : {
"remove" : {
"field" : "_ingest._value.id"
}
}
}
}
After preprocessing the result is:
{
"persons" : [
{
"name" : "John Doe"
},
{
"name" : "Jane Doe"
}
]
}
The wrapped processor can have a on_failure
definition.
For example, the id
field may not exist on all person objects.
Instead of failing the index request, you can use an on_failure
block to send the document to the 'failure_index' index for later inspection:
{
"foreach" : {
"field" : "persons",
"processor" : {
"remove" : {
"field" : "_value.id",
"on_failure" : [
{
"set" : {
"field": "_index",
"value": "failure_index"
}
}
]
}
}
}
}
In this example, if the remove
processor does fail, then
the array elements that have been processed thus far will
be updated.
Another advanced example can be found in the attachment processor documentation.
GeoIP processor
The geoip
processor adds information about the geographical location of IP addresses, based on data from the Maxmind databases.
This processor adds this information by default under the geoip
field. The geoip
processor can resolve both IPv4 and
IPv6 addresses.
The ingest-geoip
module ships by default with the GeoLite2 City, GeoLite2 Country and GeoLite2 ASN GeoIP2 databases from Maxmind made available
under the CCA-ShareAlike 4.0 license. For more details see, http://dev.maxmind.com/geoip/geoip2/geolite2/
The geoip
processor can run with other city, country and ASN GeoIP2 databases
from Maxmind. The database files must be copied into the ingest-geoip
config
directory located at $OPENSEARCH_CONFIG/ingest-geoip
. Custom database files must be
stored uncompressed and the extension must be -City.mmdb
, -Country.mmdb
, or
-ASN.mmdb
to indicate the type of the database. These database files can not
have the same filename as any of the built-in database names. The
database_file
processor option is used to specify the filename of the custom
database to use for the processor.
Using the geoip
Processor in a Pipeline
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
The field to get the ip address from for the geographical lookup. |
|
no |
geoip |
The field that will hold the geographical information looked up from the Maxmind database. |
|
no |
GeoLite2-City.mmdb |
The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the |
|
no |
[ |
Controls what properties are added to the |
|
no |
|
If |
|
no |
|
If |
*Depends on what is available in database_file
:
-
If the GeoLite2 City database is used, then the following fields may be added under the
target_field
:ip
,country_iso_code
,country_name
,continent_name
,region_iso_code
,region_name
,city_name
,timezone
,latitude
,longitude
andlocation
. The fields actually added depend on what has been found and which properties were configured inproperties
. -
If the GeoLite2 Country database is used, then the following fields may be added under the
target_field
:ip
,country_iso_code
,country_name
andcontinent_name
. The fields actually added depend on what has been found and which properties were configured inproperties
. -
If the GeoLite2 ASN database is used, then the following fields may be added under the
target_field
:ip
,asn
,organization_name
andnetwork
. The fields actually added depend on what has been found and which properties were configured inproperties
.
Here is an example that uses the default city database and adds the geographical information to the geoip
field based on the ip
field:
PUT _ingest/pipeline/geoip
{
"description" : "Add geoip info",
"processors" : [
{
"geoip" : {
"field" : "ip"
}
}
]
}
PUT my-index-00001/_doc/my_id?pipeline=geoip
{
"ip": "<IP_ADDRESS>"
}
GET my-index-00001/_doc/my_id
Which returns:
{
"found": true,
"_index": "my-index-00001",
"_type": "_doc",
"_id": "my_id",
"_version": 1,
"_seq_no": 55,
"_primary_term": 1,
"_source": {
"ip": "<IP_ADDRESS>",
"geoip": {
"continent_name": "North America",
"country_name": "United States",
"country_iso_code": "US",
"location": { "lat": 37.751, "lon": -97.822 }
}
}
}
Here is an example that uses the default country database and adds the
geographical information to the geo
field based on the ip
field. Note that
this database is included in the module. So this:
PUT _ingest/pipeline/geoip
{
"description" : "Add geoip info",
"processors" : [
{
"geoip" : {
"field" : "ip",
"target_field" : "geo",
"database_file" : "GeoLite2-Country.mmdb"
}
}
]
}
PUT my-index-00001/_doc/my_id?pipeline=geoip
{
"ip": "<IP_ADDRESS>"
}
GET my-index-00001/_doc/my_id
returns this:
{
"found": true,
"_index": "my-index-00001",
"_type": "_doc",
"_id": "my_id",
"_version": 1,
"_seq_no": 65,
"_primary_term": 1,
"_source": {
"ip": "<IP_ADDRESS>",
"geo": {
"continent_name": "North America",
"country_name": "United States",
"country_iso_code": "US",
}
}
}
Not all IP addresses find geo information from the database, When this
occurs, no target_field
is inserted into the document.
Here is an example of what documents will be indexed as when information for "<IP_ADDRESS>" cannot be found:
PUT _ingest/pipeline/geoip
{
"description" : "Add geoip info",
"processors" : [
{
"geoip" : {
"field" : "ip"
}
}
]
}
PUT my-index-00001/_doc/my_id?pipeline=geoip
{
"ip": "<IP_ADDRESS>"
}
GET my-index-00001/_doc/my_id
Which returns:
{
"_index" : "my-index-00001",
"_type" : "_doc",
"_id" : "my_id",
"_version" : 1,
"_seq_no" : 71,
"_primary_term": 1,
"found" : true,
"_source" : {
"ip" : "<IP_ADDRESS>"
}
}
Recognizing Location as a Geopoint
Although this processor enriches your document with a location
field containing
the estimated latitude and longitude of the IP address, this field will not be
indexed as a geo_point
type in OpenSearch without explicitly defining it
as such in the mapping.
You can use the following mapping for the example index above:
PUT my_ip_locations
{
"mappings": {
"properties": {
"geoip": {
"properties": {
"location": { "type": "geo_point" }
}
}
}
}
}
Node Settings
The geoip
processor supports the following setting:
ingest.geoip.cache_size
-
The maximum number of results that should be cached. Defaults to
1000
.
Note that these settings are node settings and apply to all geoip
processors, i.e. there is one cache for all defined geoip
processors.
Grok processor
Extracts structured fields out of a single text field within a document. You choose which field to extract matched fields from, as well as the grok pattern you expect will match. A grok pattern is like a regular expression that supports aliased expressions that can be reused.
This tool is perfect for syslog logs, apache and other webserver logs, mysql logs, and in general, any log format that is generally written for humans and not computer consumption. This processor comes packaged with many reusable patterns.
If you need help building patterns to match your logs, you will find the Grok Constructor is a useful tool.
Grok Basics
Grok sits on top of regular expressions, so any regular expressions are valid in grok as well. The regular expression library is Oniguruma, and you can see the full supported regexp syntax on the Oniguruma site.
Grok works by leveraging this regular expression language to allow naming existing patterns and combining them into more complex patterns that match your fields.
The syntax for reusing a grok pattern comes in three forms: %{SYNTAX:SEMANTIC}
, %{SYNTAX}
, %{SYNTAX:SEMANTIC:TYPE}
.
The SYNTAX
is the name of the pattern that will match your text. For example, 3.44
will be matched by the NUMBER
pattern and <IP_ADDRESS>
will be matched by the IP
pattern. The syntax is how you match. NUMBER
and IP
are both
patterns that are provided within the default patterns set.
The SEMANTIC
is the identifier you give to the piece of text being matched. For example, 3.44
could be the
duration of an event, so you could call it simply duration
. Further, a string <IP_ADDRESS>
might identify
the client
making a request.
The TYPE
is the type you wish to cast your named field. int
, long
, double
, float
and boolean
are supported types for coercion.
For example, you might want to match the following text:
3.44 <IP_ADDRESS>
You may know that the message in the example is a number followed by an IP address. You can match this text by using the following Grok expression.
%{NUMBER:duration} %{IP:client}
Using the Grok Processor in a Pipeline
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
The field to use for grok expression parsing |
|
yes |
- |
An ordered list of grok expression to match and extract named captures with. Returns on the first expression in the list that matches. |
|
no |
- |
A map of pattern-name and pattern tuples defining custom patterns to be used by the current processor. Patterns matching existing names will override the pre-existing definition. |
|
no |
false |
when true, |
|
no |
false |
If |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
Here is an example of using the provided patterns to extract out and name structured fields from a string field in a document.
POST _ingest/pipeline/_simulate
{
"pipeline": {
"description" : "...",
"processors": [
{
"grok": {
"field": "message",
"patterns": ["%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes:int} %{NUMBER:duration:double}"]
}
}
]
},
"docs":[
{
"_source": {
"message": "<IP_ADDRESS> GET /index.html 15824 0.043"
}
}
]
}
This pipeline will insert these named captures as new fields within the document, like so:
{
"docs": [
{
"doc": {
"_index": "_index",
"_type": "_doc",
"_id": "_id",
"_source" : {
"duration" : 0.043,
"request" : "/index.html",
"method" : "GET",
"bytes" : 15824,
"client" : "<IP_ADDRESS>",
"message" : "<IP_ADDRESS> GET /index.html 15824 0.043"
},
"_ingest": {
"timestamp": "2016-11-08T19:43:03.850+0000"
}
}
}
]
}
Custom Patterns
The Grok processor comes pre-packaged with a base set of patterns. These patterns may not always have what you are looking for. Patterns have a very basic format. Each entry has a name and the pattern itself.
You can add your own patterns to a processor definition under the pattern_definitions
option.
Here is an example of a pipeline specifying custom pattern definitions:
{
"description" : "...",
"processors": [
{
"grok": {
"field": "message",
"patterns": ["my %{FAVORITE_DOG:dog} is colored %{RGB:color}"],
"pattern_definitions" : {
"FAVORITE_DOG" : "beagle",
"RGB" : "RED|GREEN|BLUE"
}
}
}
]
}
Providing Multiple Match Patterns
Sometimes one pattern is not enough to capture the potential structure of a field. Let’s assume we
want to match all messages that contain your favorite pet breeds of either cats or dogs. One way to accomplish
this is to provide two distinct patterns that can be matched, instead of one really complicated expression capturing
the same or
behavior.
Here is an example of such a configuration executed against the simulate API:
POST _ingest/pipeline/_simulate
{
"pipeline": {
"description" : "parse multiple patterns",
"processors": [
{
"grok": {
"field": "message",
"patterns": ["%{FAVORITE_DOG:pet}", "%{FAVORITE_CAT:pet}"],
"pattern_definitions" : {
"FAVORITE_DOG" : "beagle",
"FAVORITE_CAT" : "burmese"
}
}
}
]
},
"docs":[
{
"_source": {
"message": "I love burmese cats!"
}
}
]
}
response:
{
"docs": [
{
"doc": {
"_type": "_doc",
"_index": "_index",
"_id": "_id",
"_source": {
"message": "I love burmese cats!",
"pet": "burmese"
},
"_ingest": {
"timestamp": "2016-11-08T19:43:03.850+0000"
}
}
}
]
}
Both patterns will set the field pet
with the appropriate match, but what if we want to trace which of our
patterns matched and populated our fields? We can do this with the trace_match
parameter. Here is the output of
that same pipeline, but with "trace_match": true
configured:
{
"docs": [
{
"doc": {
"_type": "_doc",
"_index": "_index",
"_id": "_id",
"_source": {
"message": "I love burmese cats!",
"pet": "burmese"
},
"_ingest": {
"_grok_match_index": "1",
"timestamp": "2016-11-08T19:43:03.850+0000"
}
}
}
]
}
In the above response, you can see that the index of the pattern that matched was "1"
. This is to say that it was the
second (index starts at zero) pattern in patterns
to match.
This trace metadata enables debugging which of the patterns matched. This information is stored in the ingest metadata and will not be indexed.
Retrieving patterns from REST endpoint
The Grok Processor comes packaged with its own REST endpoint for retrieving which patterns the processor is packaged with.
GET _ingest/processor/grok
The above request will return a response body containing a key-value representation of the built-in patterns dictionary.
{
"patterns" : {
"BACULA_CAPACITY" : "%{INT}{1,3}(,%{INT}{3})*",
"PATH" : "(?:%{UNIXPATH}|%{WINPATH})",
...
}
By default, the API returns patterns in the order they are read from disk. This sort order preserves groupings of related patterns. For example, all patterns related to parsing Linux syslog lines stay grouped together.
You can use the optional boolean s
query parameter to sort returned patterns
by key name instead.
GET _ingest/processor/grok?s
The API returns the following response.
{
"patterns" : {
"BACULA_CAPACITY" : "%{INT}{1,3}(,%{INT}{3})*",
"BACULA_DEVICE" : "%{USER}",
"BACULA_DEVICEPATH" : "%{UNIXPATH}",
...
}
This can be useful to reference as the built-in patterns change across versions.
Grok watchdog
Grok expressions that take too long to execute are interrupted and the grok processor then fails with an exception. The grok processor has a watchdog thread that determines when evaluation of a grok expression takes too long and is controlled by the following settings:
| Name | Default | Description |
|---|---|---|
|
1s |
How often to check whether there are grok evaluations that take longer than the maximum allowed execution time. |
|
1s |
The maximum allowed execution of a grok expression evaluation. |
Grok debugging
It is recommended to enable debug logging for Grok so that any additional messages may also be seen in the OpenSearch server log.
PUT _cluster/settings
{
"transient": {
"logger.org.opensearch.ingest.common.GrokProcessor": "debug"
}
}
Gsub processor
Converts a string field by applying a regular expression and a replacement. If the field is an array of string, all members of the array will be converted. If any non-string values are encountered, the processor will throw an exception.
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
The field to apply the replacement to |
|
yes |
- |
The pattern to be replaced |
|
yes |
- |
The string to replace the matching patterns with |
|
no |
|
The field to assign the converted value to, by default |
|
no |
|
If |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
{
"gsub": {
"field": "field1",
"pattern": "\\.",
"replacement": "-"
}
}
HTML strip processor
Removes HTML tags from the field. If the field is an array of strings, HTML tags will be removed from all members of the array.
|
Note
|
Each HTML tag is replaced with a \n character.
|
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
The string-valued field to remove HTML tags from |
|
no |
|
The field to assign the value to, by default |
|
no |
|
If |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
{
"html_strip": {
"field": "foo"
}
}
Join processor
Joins each element of an array into a single string using a separator character between each element. Throws an error when the field is not an array.
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
Field containing array values to join |
|
yes |
- |
The separator character |
|
no |
|
The field to assign the joined value to, by default |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
{
"join": {
"field": "joined_array_field",
"separator": "-"
}
}
JSON processor
Converts a JSON string into a structured JSON object.
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
The field to be parsed. |
|
no |
|
The field that the converted structured object will be written into. Any existing content in this field will be overwritten. |
|
no |
false |
Flag that forces the serialized json to be injected into the top level of the document. |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
All JSON-supported types will be parsed (null, boolean, number, array, object, string).
Suppose you provide this configuration of the json
processor:
{
"json" : {
"field" : "string_source",
"target_field" : "json_target"
}
}
If the following document is processed:
{
"string_source": "{\"foo\": 2000}"
}
after the json
processor operates on it, it will look like:
{
"string_source": "{\"foo\": 2000}",
"json_target": {
"foo": 2000
}
}
If the following configuration is provided, omitting the optional target_field
setting:
{
"json" : {
"field" : "source_and_target"
}
}
then after the json
processor operates on this document:
{
"source_and_target": "{\"foo\": 2000}"
}
it will look like:
{
"source_and_target": {
"foo": 2000
}
}
This illustrates that, unless it is explicitly named in the processor configuration, the target_field
is the same field provided in the required field
configuration.
KV processor
This processor helps automatically parse messages (or specific event fields) which are of the foo=bar
variety.
For example, if you have a log message which contains ip=<IP_ADDRESS> error=REFUSED
, you can parse those fields automatically by configuring:
{
"kv": {
"field": "message",
"field_split": " ",
"value_split": "="
}
}
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
The field to be parsed |
|
yes |
- |
Regex pattern to use for splitting key-value pairs |
|
yes |
- |
Regex pattern to use for splitting the key from the value within a key-value pair |
|
no |
|
The field to insert the extracted keys into. Defaults to the root of the document |
|
no |
|
List of keys to filter and insert into document. Defaults to including all keys |
|
no |
|
List of keys to exclude from document |
|
no |
|
If |
|
no |
|
Prefix to be added to extracted keys |
|
no |
|
String of characters to trim from extracted keys |
|
no |
|
String of characters to trim from extracted values |
|
no |
|
If |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
Lowercase processor
Converts a string to its lowercase equivalent. If the field is an array of strings, all members of the array will be converted.
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
The field to make lowercase |
|
no |
|
The field to assign the converted value to, by default |
|
no |
|
If |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
{
"lowercase": {
"field": "foo"
}
}
Pipeline processor
Executes another pipeline.
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
The name of the pipeline to execute. Supports template snippets. |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
{
"pipeline": {
"name": "inner-pipeline"
}
}
The name of the current pipeline can be accessed from the _ingest.pipeline
ingest metadata key.
An example of using this processor for nesting pipelines would be:
Define an inner pipeline:
PUT _ingest/pipeline/pipelineA
{
"description" : "inner pipeline",
"processors" : [
{
"set" : {
"field": "inner_pipeline_set",
"value": "inner"
}
}
]
}
Define another pipeline that uses the previously defined inner pipeline:
PUT _ingest/pipeline/pipelineB
{
"description" : "outer pipeline",
"processors" : [
{
"pipeline" : {
"name": "pipelineA"
}
},
{
"set" : {
"field": "outer_pipeline_set",
"value": "outer"
}
}
]
}
Now indexing a document while applying the outer pipeline will see the inner pipeline executed from the outer pipeline:
PUT /my-index/_doc/1?pipeline=pipelineB
{
"field": "value"
}
Response from the index request:
{
"_index": "my-index",
"_type": "_doc",
"_id": "1",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"_seq_no": 66,
"_primary_term": 1,
}
Indexed document:
{
"field": "value",
"inner_pipeline_set": "inner",
"outer_pipeline_set": "outer"
}
Remove processor
Removes existing fields. If one field doesn’t exist, an exception will be thrown.
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
Fields to be removed. Supports template snippets. |
|
no |
|
If |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
Here is an example to remove a single field:
{
"remove": {
"field": "user_agent"
}
}
To remove multiple fields, you can use the following query:
{
"remove": {
"field": ["user_agent", "url"]
}
}
Rename processor
Renames an existing field. If the field doesn’t exist or the new name is already used, an exception will be thrown.
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
The field to be renamed. Supports template snippets. |
|
yes |
- |
The new name of the field. Supports template snippets. |
|
no |
|
If |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
{
"rename": {
"field": "provider",
"target_field": "cloud.provider"
}
}
Script processor
Allows inline and stored scripts to be executed within ingest pipelines.
See How to use scripts to learn more about writing scripts. The Script Processor leverages caching of compiled scripts for improved performance. Since the script specified within the processor is potentially re-compiled per document, it is important to understand how script caching works. To learn more about caching see Script Caching.
| Name | Required | Default | Description |
|---|---|---|---|
|
no |
"painless" |
The scripting language |
|
no |
- |
The stored script id to refer to |
|
no |
- |
An inline script to be executed |
|
no |
- |
Script Parameters |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
One of id
or source
options must be provided in order to properly reference a script to execute.
You can access the current ingest document from within the script context by using the ctx
variable.
The following example sets a new field called field_a_plus_b_times_c
to be the sum of two existing
numeric fields field_a
and field_b
multiplied by the parameter param_c:
{
"script": {
"lang": "painless",
"source": "ctx.field_a_plus_b_times_c = (ctx.field_a + ctx.field_b) * params.param_c",
"params": {
"param_c": 10
}
}
}
It is possible to use the Script Processor to manipulate document metadata like _index
and _type
during
ingestion. Here is an example of an Ingest Pipeline that renames the index and type to my-index
no matter what
was provided in the original index request:
PUT _ingest/pipeline/my-index
{
"description": "use index:my-index",
"processors": [
{
"script": {
"source": """
ctx._index = 'my-index';
ctx._type = '_doc';
"""
}
}
]
}
Using the above pipeline, we can attempt to index a document into the any-index
index.
PUT any-index/_doc/1?pipeline=my-index
{
"message": "text"
}
The response from the above index request:
{
"_index": "my-index",
"_type": "_doc",
"_id": "1",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"_seq_no": 89,
"_primary_term": 1,
}
In the above response, you can see that our document was actually indexed into my-index
instead of
any-index
. This type of manipulation is often convenient in pipelines that have various branches of transformation,
and depending on the progress made, indexed into different indices.
Set processor
Sets one field and associates it with the specified value. If the field already exists, its value will be replaced with the provided one.
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
The field to insert, upsert, or update. Supports template snippets. |
|
yes |
- |
The value to be set for the field. Supports template snippets. |
|
no |
true |
If processor will update fields with pre-existing non-null-valued field. When set to |
|
no |
|
If |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
{
"description" : "sets the value of count to 1"
"set": {
"field": "count",
"value": 1
}
}
This processor can also be used to copy data from one field to another. For example:
PUT _ingest/pipeline/set_os
{
"description": "sets the value of host.os.name from the field os",
"processors": [
{
"set": {
"field": "host.os.name",
"value": "{{os}}"
}
}
]
}
POST _ingest/pipeline/set_os/_simulate
{
"docs": [
{
"_source": {
"os": "Ubuntu"
}
}
]
}
Result:
{
"docs" : [
{
"doc" : {
"_index" : "_index",
"_type" : "_doc",
"_id" : "_id",
"_source" : {
"host" : {
"os" : {
"name" : "Ubuntu"
}
},
"os" : "Ubuntu"
},
"_ingest" : {
"timestamp" : "2019-03-11T21:54:37.909224Z"
}
}
}
]
}
Set security user processor
Sets user-related details (such as username
, roles
, email
, full_name
,
metadata
, api_key
, realm
and authentication_type
) from the current
authenticated user to the current document by pre-processing the ingest.
The api_key
property exists only if the user authenticates with an
API key. It is an object containing the id
and name
fields of the API key.
The realm
property is also an object with two fields, name
and type
.
When using API key authentication, the realm
property refers to the realm
from which the API key is created.
The authentication_type
property is a string that can take value from
REALM
, API_KEY
, TOKEN
and ANONYMOUS
.
|
Important
|
Requires an authenticated user for the index request. |
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
The field to store the user information into. |
|
no |
[ |
Controls what user related properties are added to the |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
The following example adds all user details for the current authenticated user
to the user
field for all documents that are processed by this pipeline:
{
"processors" : [
{
"set_security_user": {
"field": "user"
}
}
]
}
Sort processor
Sorts the elements of an array ascending or descending. Homogeneous arrays of numbers will be sorted numerically, while arrays of strings or heterogeneous arrays of strings + numbers will be sorted lexicographically. Throws an error when the field is not an array.
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
The field to be sorted |
|
no |
|
The sort order to use. Accepts |
|
no |
|
The field to assign the sorted value to, by default |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
{
"sort": {
"field": "array_field_to_sort",
"order": "desc"
}
}
Split processor
Splits a field into an array using a separator character. Only works on string fields.
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
The field to split |
|
yes |
- |
A regex which matches the separator, eg |
|
no |
|
The field to assign the split value to, by default |
|
no |
|
If |
|
no |
|
Preserves empty trailing fields, if any. |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
{
"split": {
"field": "my_field",
"separator": "\\s+" (1)
}
}
-
Treat all consecutive whitespace characters as a single separator
If the preserve_trailing
option is enabled, any trailing empty fields in the input will be preserved. For example,
in the configuration below, a value of A,,B,,
in the my_field
property will be split into an array of five elements
["A", "", "B", "", ""]
with two empty trailing fields. If the preserve_trailing
property were not enabled, the two
empty trailing fields would be discarded resulting in the three-element array ["A", "", "B"]
.
{
"split": {
"field": "my_field",
"separator": ",",
"preserve_trailing": true
}
}
Trim processor
Trims whitespace from field. If the field is an array of strings, all members of the array will be trimmed.
|
Note
|
This only works on leading and trailing whitespace. |
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
The string-valued field to trim whitespace from |
|
no |
|
The field to assign the trimmed value to, by default |
|
no |
|
If |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
{
"trim": {
"field": "foo"
}
}
Uppercase processor
Converts a string to its uppercase equivalent. If the field is an array of strings, all members of the array will be converted.
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
The field to make uppercase |
|
no |
|
The field to assign the converted value to, by default |
|
no |
|
If |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
{
"uppercase": {
"field": "foo"
}
}
URL decode processor
URL-decodes a string. If the field is an array of strings, all members of the array will be decoded.
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
The field to decode |
|
no |
|
The field to assign the converted value to, by default |
|
no |
|
If |
|
no |
- |
Conditionally execute this processor. |
|
no |
- |
Handle failures for this processor. See Handling Failures in Pipelines. |
|
no |
|
Ignore failures for this processor. See Handling Failures in Pipelines. |
|
no |
- |
An identifier for this processor. Useful for debugging and metrics. |
{
"urldecode": {
"field": "my_url_to_decode"
}
}
User agent processor
The user_agent
processor extracts details from the user agent string a browser sends with its web requests.
This processor adds this information by default under the user_agent
field.
The ingest-user-agent module ships by default with the regexes.yaml made available by uap-java with an Apache 2.0 license. For more details see https://github.com/ua-parser/uap-core.
Using the user_agent Processor in a Pipeline
| Name | Required | Default | Description |
|---|---|---|---|
|
yes |
- |
The field containing the user agent string. |
|
no |
user_agent |
The field that will be filled with the user agent details. |
|
no |
- |
The name of the file in the |
|
no |
[ |
Controls what properties are added to |
|
no |
|
If |
|
no |
|
Whether to return the output in Elastic Common Schema format. NOTE: This setting is deprecated and will be removed in a future version. |
Here is an example that adds the user agent details to the user_agent
field based on the agent
field:
PUT _ingest/pipeline/user_agent
{
"description" : "Add user agent information",
"processors" : [
{
"user_agent" : {
"field" : "agent"
}
}
]
}
PUT my-index-000001/_doc/my_id?pipeline=user_agent
{
"agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"
}
GET my-index-000001/_doc/my_id
Which returns
{
"found": true,
"_index": "my-index-000001",
"_type": "_doc",
"_id": "my_id",
"_version": 1,
"_seq_no": 22,
"_primary_term": 1,
"_source": {
"agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36",
"user_agent": {
"name": "Chrome",
"original": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36",
"version": "51.0.2704.103",
"os": {
"name": "Mac OS X",
"version": "10.10.5",
"full": "Mac OS X 10.10.5"
},
"device" : {
"name" : "Mac"
},
}
}
}
Using a custom regex file
To use a custom regex file for parsing the user agents, that file has to be put into the config/ingest-user-agent
directory and
has to have a .yml
filename extension. The file has to be present at node startup, any changes to it or any new files added
while the node is running will not have any effect.
In practice, it will make most sense for any custom regex file to be a variant of the default file, either a more recent version or a customised version.
The default file included in ingest-user-agent
is the regexes.yaml
from uap-core: https://github.com/ua-parser/uap-core/blob/master/regexes.yaml
Node Settings
The user_agent
processor supports the following setting:
ingest.user_agent.cache_size
-
The maximum number of results that should be cached. Defaults to
1000
.
Note that these settings are node settings and apply to all user_agent
processors, i.e. there is one cache for all defined user_agent
processors.