Creating Robust JMS Applications
This section explains how to use features of the JMS API to achieve the level of reliability and performance your application requires. Many people choose to implement JMS applications because they cannot tolerate dropped or duplicate messages and require that every message be received once and only once. The JMS API provides this functionality.
The most reliable way to produce a message is to send a PERSISTENT message within a transaction. JMS messages are PERSISTENT by default. A transaction is a unit of work into which you can group a series of operations, such as message sends and receives, so that the operations either all succeed or all fail. For details, see Specifying Message Persistence and Using JMS API Local Transactions.
The most reliable way to consume a message is to do so within a transaction, either from a queue or from a durable subscription to a topic. For details, see Creating Temporary Destinations, Creating Durable Subscriptions, and Using JMS API Local Transactions.
For other applications, a lower level of reliability can reduce overhead and improve performance. You can send messages with varying priority levels (see Setting Message Priority Levels) and you can set them to expire after a certain length of time (see Allowing Messages to Expire).
The JMS API provides several ways to achieve various kinds and degrees of reliability. This section divides them into two categories, basic and advanced.
The following sections describe these features as they apply to JMS clients. Some of the features work differently in Java EE applications; in these cases, the differences are noted here and are explained in detail in Using the JMS API in Java EE Applications.
Using Basic Reliability Mechanisms
The basic mechanisms for achieving or affecting reliable message delivery are as follows:
Controlling message acknowledgment: You can specify various levels of control over message acknowledgment.
Specifying message persistence: You can specify that messages are persistent, meaning that they must not be lost in the event of a provider failure.
Setting message priority levels: You can set various priority levels for messages, which can affect the order in which the messages are delivered.
Allowing messages to expire: You can specify an expiration time for messages so that they will not be delivered if they are obsolete.
Creating temporary destinations: You can create temporary destinations that last only for the duration of the connection in which they are created.
Controlling Message Acknowledgment
Until a JMS message has been acknowledged, it is not considered to be successfully consumed. The successful consumption of a message ordinarily takes place in three stages.
The client receives the message.
The client processes the message.
The message is acknowledged. Acknowledgment is initiated either by the JMS provider or by the client, depending on the session acknowledgment mode.
In transacted sessions (see Using JMS API Local Transactions), acknowledgment happens automatically when a transaction is committed. If a transaction is rolled back, all consumed messages are redelivered.
In nontransacted sessions, when and how a message is acknowledged depend on the value specified as the second argument of the createSession method. The three possible argument values are as follows:
Session.AUTO_ACKNOWLEDGE: The session automatically acknowledges a client’s receipt of a message either when the client has successfully returned from a call to receive or when the MessageListener it has called to process the message returns successfully. A synchronous receive in an AUTO_ACKNOWLEDGE session is the one exception to the rule that message consumption is a three-stage process as described earlier.
In this case, the receipt and acknowledgment take place in one step, followed by the processing of the message.
Session.CLIENT_ACKNOWLEDGE: A client acknowledges a message by calling the message’s acknowledge method. In this mode, acknowledgment takes place on the session level: Acknowledging a consumed message automatically acknowledges the receipt of all messages that have been consumed by its session. For example, if a message consumer consumes ten messages and then acknowledges the fifth message delivered, all ten messages are acknowledged.
Session.DUPS_OK_ACKNOWLEDGE: This option instructs the session to lazily acknowledge the delivery of messages. This is likely to result in the delivery of some duplicate messages if the JMS provider fails, so it should be used only by consumers that can tolerate duplicate messages. (If the JMS provider redelivers a message, it must set the value of the JMSRedelivered message header to true.) This option can reduce session overhead by minimizing the work the session does to prevent duplicates.
If messages have been received from a queue but not acknowledged when a session terminates, the JMS provider retains them and redelivers them when a consumer next accesses the queue. The provider also retains unacknowledged messages for a terminated session that has a durable TopicSubscriber. (See Creating Durable Subscriptions.) Unacknowledged messages for a nondurable TopicSubscriber are dropped when the session is closed.
If you use a queue or a durable subscription, you can use the Session.recover method to stop a nontransacted session and restart it with its first unacknowledged message. In effect, the session’s series of delivered messages is reset to the point after its last acknowledged message. The messages it now delivers may be different from those that were originally delivered, if messages have expired or if higher-priority messages have arrived. For a nondurable TopicSubscriber, the provider may drop unacknowledged messages when its session is recovered.
The sample program in XREF the next section demonstrates two ways to ensure that a message will not be acknowledged until processing of the message is complete.
Specifying Message Persistence
The JMS API supports two delivery modes for messages to specify whether messages are lost if the JMS provider fails. These delivery modes are fields of the DeliveryMode interface.
The PERSISTENT delivery mode, which is the default, instructs the JMS provider to take extra care to ensure that a message is not lost in transit in case of a JMS provider failure. A message sent with this delivery mode is logged to stable storage when it is sent.
The NON_PERSISTENT delivery mode does not require the JMS provider to store the message or otherwise guarantee that it is not lost if the provider fails.
You can specify the delivery mode in either of two ways.
You can use the setDeliveryMode method of the MessageProducer interface to set the delivery mode for all messages sent by that producer. For example, the following call sets the delivery mode to NON_PERSISTENT for a producer:
producer.setDeliveryMode(DeliveryMode.NON_PERSISTENT);
You can use the long form of the send or the publish method to set the delivery mode for a specific message. The second argument sets the delivery mode. For example, the following send call sets the delivery mode for message to NON_PERSISTENT:
producer.send(message, DeliveryMode.NON_PERSISTENT, 3, 10000);
The third and fourth arguments set the priority level and expiration time, which are described in the next two subsections.
If you do not specify a delivery mode, the default is PERSISTENT. Using the NON_PERSISTENT delivery mode may improve performance and reduce storage overhead, but you should use it only if your application can afford to miss messages.
Setting Message Priority Levels
You can use message priority levels to instruct the JMS provider to deliver urgent messages first. You can set the priority level in either of two ways.
You can use the setPriority method of the MessageProducer interface to set the priority level for all messages sent by that producer. For example, the following call sets a priority level of 7 for a producer:
producer.setPriority(7);
You can use the long form of the send or the publish method to set the priority level for a specific message. The third argument sets the priority level. For example, the following send call sets the priority level for message to 3:
producer.send(message, DeliveryMode.NON_PERSISTENT, 3, 10000);
The ten levels of priority range from 0 (lowest) to 9 (highest). If you do not specify a priority level, the default level is 4. A JMS provider tries to deliver higher-priority messages before lower-priority ones but does not have to deliver messages in exact order of priority.
Allowing Messages to Expire
By default, a message never expires. If a message will become obsolete after a certain period, however, you may want to set an expiration time. You can do this in either of two ways.
You can use the setTimeToLive method of the MessageProducer interface to set a default expiration time for all messages sent by that producer. For example, the following call sets a time to live of one minute for a producer:
producer.setTimeToLive(60000);
You can use the long form of the send or the publish method to set an expiration time for a specific message. The fourth argument sets the expiration time in milliseconds. For example, the following send call sets a time to live of 10 seconds:
producer.send(message, DeliveryMode.NON_PERSISTENT, 3, 10000);
If the specified timeToLive value is 0, the message never expires.
When the message is sent, the specified timeToLive is added to the current time to give the expiration time. Any message not delivered before the specified expiration time is destroyed. The destruction of obsolete messages conserves storage and computing resources.
Creating Temporary Destinations
Normally, you create JMS destinations (queues and topics) administratively rather than programmatically. Your JMS provider includes a tool that you use to create and remove destinations, and it is common for destinations to be long-lasting.
The JMS API also enables you to create destinations (TemporaryQueue and TemporaryTopic objects) that last only for the duration of the connection in which they are created. You create these destinations dynamically using the Session.createTemporaryQueue and the Session.createTemporaryTopic methods.
The only message consumers that can consume from a temporary destination are those created by the same connection that created the destination. Any message producer can send to the temporary destination. If you close the connection that a temporary destination belongs to, the destination is closed and its contents are lost.
You can use temporary destinations to implement a simple request/reply mechanism. If you create a temporary destination and specify it as the value of the JMSReplyTo message header field when you send a message, then the consumer of the message can use the value of the JMSReplyTo field as the destination to which it sends a reply. The consumer can also reference the original request by setting the JMSCorrelationID header field of the reply message to the value of the JMSMessageID header field of the request. For example, an onMessage method can create a session so that it can send a reply to the message it receives. It can use code such as the following:
producer = session.createProducer(msg.getJMSReplyTo()); replyMsg = session.createTextMessage("Consumer " + "processed message: " + msg.getText()); replyMsg.setJMSCorrelationID(msg.getJMSMessageID()); producer.send(replyMsg);
For more examples, see Chapter 48, Java Message Service Examples.
Using Advanced Reliability Mechanisms
The more advanced mechanisms for achieving reliable message delivery are the following:
Creating durable subscriptions: You can create durable topic subscriptions, which receive messages published while the subscriber is not active. Durable subscriptions offer the reliability of queues to the publish/subscribe message domain.
Using local transactions: You can use local transactions, which allow you to group a series of sends and receives into an atomic unit of work. Transactions are rolled back if they fail at any time.
Creating Durable Subscriptions
To ensure that a pub/sub application receives all published messages, use PERSISTENT delivery mode for the publishers. In addition, use durable subscriptions for the subscribers.
The Session.createConsumer method creates a nondurable subscriber if a topic is specified as the destination. A nondurable subscriber can receive only messages that are published while it is active.
At the cost of higher overhead, you can use the Session.createDurableSubscriber method to create a durable subscriber. A durable subscription can have only one active subscriber at a time.
A durable subscriber registers a durable subscription by specifying a unique identity that is retained by the JMS provider. Subsequent subscriber objects that have the same identity resume the subscription in the state in which it was left by the preceding subscriber. If a durable subscription has no active subscriber, the JMS provider retains the subscription’s messages until they are received by the subscription or until they expire.
You establish the unique identity of a durable subscriber by setting the following:
A client ID for the connection
A topic and a subscription name for the subscriber
You set the client ID administratively for a client-specific connection factory using either the command line or the Administration Console.
After using this connection factory to create the connection and the session, you call the createDurableSubscriber method with two arguments: the topic and a string that specifies the name of the subscription:
String subName = "MySub"; MessageConsumer topicSubscriber = session.createDurableSubscriber(myTopic, subName);
The subscriber becomes active after you start the Connection or TopicConnection. Later, you might close the subscriber:
topicSubscriber.close();
The JMS provider stores the messages sent or published to the topic, as it would store messages sent to a queue. If the program or another application calls createDurableSubscriber using the same connection factory and its client ID, the same topic, and the same subscription name, the subscription is reactivated, and the JMS provider delivers the messages that were published while the subscriber was inactive.
To delete a durable subscription, first close the subscriber, and then use the unsubscribe method, with the subscription name as the argument:
topicSubscriber.close(); session.unsubscribe("MySub");
The unsubscribe method deletes the state that the provider maintains for the subscriber.
Figure 47-6 and Figure 47-7 show the difference between a nondurable and a durable subscriber. With an ordinary, nondurable subscriber, the subscriber and the subscription begin and end at the same point and are, in effect, identical. When a subscriber is closed, the subscription also ends. Here, create stands for a call to Session.createConsumer with a Topic argument, and close stands for a call to MessageConsumer.close. Any messages published to the topic between the time of the first close and the time of the second create are not consumed by the subscriber. In Figure 47-6, the subscriber consumes messages M1, M2, M5, and M6, but messages M3 and M4 are lost.
With a durable subscriber, the subscriber can be closed and re-created, but the subscription continues to exist and to hold messages until the application calls the unsubscribe method. In Figure 47-7, create stands for a call to Session.createDurableSubscriber, close stands for a call to MessageConsumer.close, and unsubscribe stands for a call to Session.unsubscribe. Messages published while the subscriber is closed are received when the subscriber is created again. So even though messages M2, M4, and M5 arrive while the subscriber is closed, they are not lost.
See A Message Acknowledgment Example, A Durable Subscription Example, and An Application That Uses the JMS API with a Session Bean for examples of Java EE applications that use durable subscriptions.
Using JMS API Local Transactions
You can group a series of operations into an atomic unit of work called a transaction. If any one of the operations fails, the transaction can be rolled back, and the operations can be attempted again from the beginning. If all the operations succeed, the transaction can be committed.
In a JMS client, you can use local transactions to group message sends and receives. The JMS API Session interface provides commit and rollback methods that you can use in a JMS client.  A transaction rollback means that all produced messages are destroyed and all consumed messages are recovered and redelivered unless they have expired (see Allowing Messages to Expire).
A transacted session is always involved in a transaction. As soon as the commit or the rollback method is called, one transaction ends and another transaction begins. Closing a transacted session rolls back its transaction in progress, including any pending sends and receives.
In an Enterprise JavaBeans component, you cannot use the Session.commit and Session.rollback methods. Instead, you use distributed transactions, which are described in Using the JMS API in Java EE Applications.
You can combine several sends and receives in a single JMS API local transaction. If you do so, you need to be careful about the order of the operations. You will have no problems if the transaction consists of all sends or all receives or if the receives come before the sends. But if you try to use a request/reply mechanism, whereby you send a message and then try to receive a reply to the sent message in the same transaction, the program will hang, because the send cannot take place until the transaction is committed. The following code fragment illustrates the problem:
// Don’t do this! outMsg.setJMSReplyTo(replyQueue); producer.send(outQueue, outMsg); consumer = session.createConsumer(replyQueue); inMsg = consumer.receive(); session.commit();
Because a message sent during a transaction is not actually sent until the transaction is committed, the transaction cannot contain any receives that depend on that message’s having been sent.
In addition, the production and the consumption of a message cannot both be part of the same transaction. The reason is that the transactions take place between the clients and the JMS provider, which intervenes between the production and the consumption of the message. Figure 47-8 illustrates this interaction.
The sending of one or more messages to one or more destinations by client 1 can form a single transaction, because it forms a single set of interactions with the JMS provider using a single session. Similarly, the receiving of one or more messages from one or more destinations by client 2 also forms a single transaction using a single session. But because the two clients have no direct interaction and are using two different sessions, no transactions can take place between them.
Another way of putting this is that the act of producing and/or consuming messages in a session can be transactional, but the act of producing and consuming a specific message across different sessions cannot be transactional.
This is the fundamental difference between messaging and synchronized processing. Instead of tightly coupling the sending and receiving of data, message producers and consumers use an alternative approach to reliability, one that is built on a JMS provider’s ability to supply a once-and-only-once message delivery guarantee.
When you create a session, you specify whether it is transacted. The first argument to the createSession method is a boolean value. A value of true means that the session is transacted; a value of false means that it is not transacted. The second argument to this method is the acknowledgment mode, which is relevant only to nontransacted sessions (see Controlling Message Acknowledgment). If the session is transacted, the second argument is ignored, so it is a good idea to specify 0 to make the meaning of your code clear. For example:
session = connection.createSession(true, 0);
The commit and the rollback methods for local transactions are associated with the session. You can combine queue and topic operations in a single transaction if you use the same session to perform the operations. For example, you can use the same session to receive a message from a queue and send a message to a topic in the same transaction.
You can pass a client program’s session to a message listener’s constructor function and use it to create a message producer. In this way, you can use the same session for receives and sends in asynchronous message consumers.
A Local Transaction Example provides an example of the use of JMS API local transactions.


Your friend comes with you. You live your life happily, but you live in fear as well, because the fishermen come daily to the lake. Fortunately, you and your friend escape whenever they come.


| Game 1 (1-50) Cho Chikun (B) vs Yoda Norimoto (W), 1995-01-12 |
Game 2 (1-50) Cho Chikun (B) vs Kobayashi Satoru (W), 1995-01-18,19 |
| Game 3 (1-50) Cho Chikun (B) vs Kato Masao (W), 1995-01-26 |
Game 4 (1-50) Kobayashi Satoru (B) vs Cho Chikun (W), 1995-02-01,02 |


Hello!
Your Keenthemes password was just changed.
If you didn't change your password, please contact our
support team. Your security is very important to us!
Kind regards,
The Keenthemes Team.
|
Floor 10, 101 Avenue of the Light Square, New York, NY, 10050.
Copyright ©
Keenthemes.
|
|


|
| void | start (string label) |
| |
| void | stop (string label) |
| |
| bool | waiting (string label) |
| |
| bool | busy (string label) |
| |
| bool | finished (string label) |
| |
| void | remove (string label) |
| |
| void | removeAllEffects () |
| |
| void | makeTween (string label, float start_value, float end_value, float duration_in_seconds) |
| |
| float | tween (string label, int style) |
| |
| void | makeShake (string label, int shake_size, float duration_in_seconds) |
| |
| float | shake (string label) |
| |
| void | makeOscillation (string label, float oscillation_size, float period_in_seconds) |
| |
| float | oscillation (string label) |
| |
◆ TWEEN_STYLES
Effects Styles.
See graphs in http://www.friendsonmountains.com/blog/2018/07/14/lets-make-honey-version-0-07-effects
| Enumerator |
|---|
| LINEAR | Linear Tween. Constant speed.
|
| SIGMOID | Sigmoid Tween. Starts slow, gets fast in the middle, ends slow.
|
| CUBIC | Cubic Tween. Starts fast, pauses in the middle, ends fast.
|
| RAMPDOWN | Rampdown Tween. Starts fast, ends slow.
|
| RAMPUP | Rampdown Tween. Starts slow, ends fast.
|
| SINEWAVE | Sinewave Tween. Moves from start to end, then boomerangs back to start.
|
◆ instance()
| Effects & Honey::Effects::instance |
( |
| ) |
|
|
static |
Singleton instance getter.
- Returns
- This'll get you the one instance of Effects you're supposed to have.
◆ start()
| void Honey::Effects::start |
( |
string |
label | ) |
|
Start the effect!
- Parameters
-
| label | The effect you want to start. |
◆ stop()
| void Honey::Effects::stop |
( |
string |
label | ) |
|
Stop the effect!
- Parameters
-
| label | The effect you want to stop. |
◆ waiting()
| bool Honey::Effects::waiting |
( |
string |
label | ) |
|
Check whether an effect is waiting.
- Parameters
-
| label | The effect you want to check. |
- Returns
- true if and only if the label exists and the state is waiting.
◆ busy()
| bool Honey::Effects::busy |
( |
string |
label | ) |
|
Check whether an effect is busy.
- Parameters
-
| label | The effect you want to check. |
- Returns
- true if and only if the label exists and the state is busy.
◆ finished()
| bool Honey::Effects::finished |
( |
string |
label | ) |
|
Check whether an effect is finished.
- Parameters
-
| label | The effect you want to check. |
- Returns
- true if and only if the label exists and the state is finished.
◆ remove()
| void Honey::Effects::remove |
( |
string |
label | ) |
|
Remove an effect.
- Parameters
-
| label | The effect you want to remove. |
◆ removeAllEffects()
| void Honey::Effects::removeAllEffects |
( |
| ) |
|
Remove all effects currently stored in the Effects instance.
◆ makeTween()
| void Honey::Effects::makeTween |
( |
string |
label, |
|
|
float |
start_value, |
|
|
float |
end_value, |
|
|
float |
duration_in_seconds |
|
) |
| |
Create a tween which interpolates between two values over time.
- Parameters
-
| label | Name of the effect to create. |
| start_value | The value when the tween starts. |
| end_value | The value when the tween ends. |
| duration_in_seconds | The duration of the tween effect. |
◆ tween()
| float Honey::Effects::tween |
( |
string |
label, |
|
|
int |
style |
|
) |
| |
Get the interpolated value of a given tween effect.
Note that time is calculated for you in the background.
- Parameters
-
| label | Name of the effect. |
| style | Tween style (see TWEEN_STYLES enum for different styles). |
- Returns
- A value somewhere between the start and end values for this tween.
◆ makeShake()
| void Honey::Effects::makeShake |
( |
string |
label, |
|
|
int |
shake_size, |
|
|
float |
duration_in_seconds |
|
) |
| |
Create a shake effect which adds some random noise.
- Parameters
-
| label | Name of the effect to create. |
| shake_size | Shake noise is sampled from the range (-shake_size, shake_size). |
| duration_in_seconds | The duration of the shake effect. |
◆ shake()
| float Honey::Effects::shake |
( |
string |
label | ) |
|
Grab a value from the random shaker.
This value will range from -shake_size to shake_size.
- Parameters
-
- Returns
- A value in the range (-shake_size, shake_size) as defined in makeShake.
◆ makeOscillation()
| void Honey::Effects::makeOscillation |
( |
string |
label, |
|
|
float |
oscillation_size, |
|
|
float |
period_in_seconds |
|
) |
| |
Create an oscillation effect, which goes back and forth in a sine wave.
- Parameters
-
| label | Name of the effect to create. |
| oscillation_size | Effect ranges from (-oscillation_size, oscillation_size). |
| period_in_seconds | A full oscillation is completed every period_in_seconds. |
◆ oscillation()
| float Honey::Effects::oscillation |
( |
string |
label | ) |
|
Grab a value from the random shaker.
This value will range from -oscillation_size to oscillation_size.
Note that time is calculated for you in the background.
- Parameters
-
- Returns
- A value in the range (-oscillation_size, oscillation_size) as defined in makeOscillation.
The documentation for this class was generated from the following files:
- /Users/matthewcarlin/Projects/2018_Honey/Engine/Source/effects.h
- /Users/matthewcarlin/Projects/2018_Honey/Engine/Source/effects.cpp


--- layout: archive title: Zulip Chat Archive permalink: /stream/113488-general/topic/magical.20restart.20fix.html ---
This is peculiar. Every fifth lean server restart or so (which I need to do fairly often due to "excessive memory consumption" thing), one of my theorems breaks and can be unbroken by restarting Lean again. I have been failing to make a minimal example though - seems rather arbitrary.
I've been experiencing the same sort of thing.
{% endraw %}Last updated: Apr 24 2020 at 19:02 UTC


Click on dismiss to cancel the alarm, or delete to dismiss and remove the alarm


This is the base class for strategy plugins, which contains some
common code useful to all strategies like running handlers, cleanup
actions, etc.
|
|
|
|
|
| run(self,
iterator,
play_context,
result=True) |
source code
|
|
|
|
|
|
|
|
|
|
add_tqm_variables(self,
vars,
play)
Base class method to add extra variables/information to the list of
task vars sent through the executor engine regarding the task queue
manager state. |
source code
|
|
|
|
_queue_task(self,
host,
task,
task_vars,
play_context)
handles queueing the task up to be sent to a worker |
source code
|
|
|
|
_process_pending_results(self,
iterator,
one_pass=False)
Reads results off the final queue and takes appropriate action based
on the result (executing callbacks, updating state, etc.). |
source code
|
|
|
|
_wait_on_pending_results(self,
iterator)
Wait for the shared counter to drop to zero, using a short sleep
between checks to ensure we don't spin lock |
source code
|
|
|
|
_add_host(self,
host_info,
iterator)
Helper function to add a new host to inventory based on a task
result. |
source code
|
|
|
|
_add_group(self,
host,
result_item)
Helper function to add a group (if it does not exist), and to assign
the specified host to that group. |
source code
|
|
|
|
_load_included_file(self,
included_file,
iterator,
is_handler=False)
Loads an included YAML file of tasks, applying the optional set of
variables. |
source code
|
|
|
|
run_handlers(self,
iterator,
play_context)
Runs handlers on those hosts which have been notified. |
source code
|
|
|
|
| _do_handler_run(self,
handler,
handler_name,
iterator,
play_context,
notified_hosts=None) |
source code
|
|
|
|
|
|
|
| _execute_meta(self,
task,
play_context,
iterator) |
source code
|
|
|
Inherited from object :
__delattr__ ,
__format__ ,
__getattribute__ ,
__hash__ ,
__new__ ,
__reduce__ ,
__reduce_ex__ ,
__repr__ ,
__setattr__ ,
__sizeof__ ,
__str__ ,
__subclasshook__
|


The One component is a source for the value one (true). Anything it is connected to will receive a constant one value.


Gelb’s contract as Met Opera head extended through 2026-27
by Associated PressNEW YORK (AP) — Peter Gelb is set to become the Metropolitan Opera’s third-longest tenured general manager.
The company’s board of directors approved a five-year contract last Thursday that extends Gelb’s term through the 2026-27 season.
Gelb, who turned 66 on Nov. 10, succeeded Joseph Volpe for the 2006-07 season. If Gelb serves through his new contract, only Giulio Gatti-Casazza (1908-35) and Rudolf Bing (1950-72) will have run the Met for the longer periods.
Gelb started simulcasts to movie theaters around the world and this season instituted regular Sunday matinees of staged operas for the first time at the Met. The company will start splitting its season in 2020-21, going dark in February and extending its schedule into June. Ticket sales slumped following the Great Recession but have stabilized in recent years, when the company has run at a small deficit.


Please select a folder name for your WOOL project, and
choose where the project should be stored.
Project name
Project directory
Create new project
1. Project Location
2. Languages
3. Finished!
Select which language will be used by default in this
project. You can enter the ISO code for the language, and
optionally
region, or select a code from a basic or full language set.
Default Language:
Create new project
1. Project Location
2. Languages
3. Finished!
Error
Cannot initialize project because an error occurred.


Ocean [ak75], ocean, in Crystal Sea
Last updated Turn #66
Ocean [ak75], ocean, in Crystal Sea
------------------------------------------------------------------------
22: Orion I [hd3f], ship, 2 hulls, 1 rowing port, 6 sails, 19% damaged,
22: arrived from Endless Plains [ak74], owner:
22: Phantasm [n4b], Imperial Empire, with 12 sailors, one ox
23: Orion I [hd3f], ship, 2 hulls, 1 rowing port, 6 sails, 19% damaged,
23: departed for Ocean [ak76].
30: The fog has cleared.
Routes leaving Ocean:
North, to Ocean [aj75], 6 days
East, to Ocean [ak76], 6 days
South, plain, to Endless Plains [al75], Imperia, 2 days
West, plain, to Endless Plains [ak74], Imperia, 2 days


{% extends 'paths/base.html' %} {% load static %} {% block title_block %} Previous Searches - Vilo Sky {% endblock %} {% block body_block %}
Hi {{ user.first_name }} {{ user.last_name }}, welcome to your previous searches! From here you can view all of your previous searches containing our advice for you.
You have not run any searches. You can run one by heading to Find Paths.
{% endif %}


WATCH: Ndlovu Youth Choir's performances at The Eat Out Mercedes-Benz Restaurant Awards gave us goosebumps!
Cape Town - The top restaurants in South Africa were announced at the 2019 Eat Out Mercedes-Benz Restaurant Awards on Sunday 17 November at GrandWest in Cape Town and we were there to capture all the best musical moments!
The best of the best in the industry were celebrated in a spectacular display of gourmet food, creative drinks and top-notch entertainment which had over 700 guests, including some of SA’s favourite celebrities, on their feet. The Ndlovu Youth Choir were there to entertain the masses and we captured our favourite performances from their amazing rendition of Vicky Sampson's African Dream to Toto's hit song Africa.
WATCH THEIR PERFORMANCES HERE:
The countdown from 20 to 11 revealed the names to watch out for, prior to the announcement of the much-anticipated Top 10 restaurants.
Last year, The Restaurant at Waterkloof halted The Test Kitchen’s six-year streak when they walked away as Restaurant of the Year – however this year, it’s La Colombe that walks away as the new number one! The Test Kitchen remains at number 2 for a second year and The Restaurant at Waterkloof is now at number 3.
The 2019 Eat Out S.Pellegrino & Acqua Panna Chef of the Year is Bertus Basson of Overture and Eike. Bertus not only keeps colourful, true South African cuisine alive; he continuously sets the bar higher for himself and his team. Bertus tells great South African stories through his food – on all levels.
READ NEXT: South Africa’s top 20 restaurants for 2019.
The Eat Out Service Excellence Award went to Test Kitchen, for the second year in a row. Joseph Dhafana of La Colombe was named winner of the Eat Out Wine Service Award; a gentle and humble man with exceptional wine knowledge. FYN was recognised with the Eat Out VISI Style Award, while Jewell’s Restaurant walked away with the Eat Out Retail Capital New Restaurant of the Year Award. Jennifer Hugé was recognised with the Eat Out John Psillos Award for Outstanding Contribution to Service.
The winning restaurants were determined by the scores of a panel of anonymous judges, headed up by Chief Judge Margot Janse, who dined around the country unannounced and scored the restaurants against the rigorous judging criteria in line with international standards. Although the winners have been announced, the panel will remain anonymous. Not only does this allow the judges to remain on the panel for the coming year(s) for consistency purposes, but the anonymous judging also ensures that the judges do not get special treatment – they need to receive the same experience as any other diner.
The chefs of the 30 nominated restaurants were given the opportunity to vote for a champion chef amongst their peers and it was Ivor Jones of Chefs Warehouse at Beau Constantia that was awarded the Eat Out Graham Beck Chefs' Chef Award. The Eat Out Nederburg Rising Star Award was handed to Katlego Mlambo of The Marabi Club.
Annette Kesler was honoured with the Eat Out Lannice Snyman Lifetime Achievement Award. A doyenne of South African food writers, Kesler has had multiple roles as food editor, cookbook author and creator of Showcook, along with mentoring many young chefs over the past several decades.
Camphors at Vergelegen received the Eat Out Woolworths Sustainability Award in recognition of its commitment to implementing environmentally and socially responsible practices. The winner was selected by a separate panel of judges, consisting of Sonia Mountford of Eategrity, Karen Welter of the Longtable Project and Pavitray Pillay of WWF-SASSI.
In an exciting turn of events, three new awards were introduced in 2019. The Eat Out Nespresso Innovation Award was awarded to The Test Kitchen in celebration of their ingenuity and originality of their pub pop-up. Ishay Govender-Ypma and Zayaan Khan of the SA POC at the Table initiative received the Eat Out Bridget McCarney Gamechanger Award in recognition of their efforts in creating positive change in how the industry operates. 2019 also saw Dorah Sitole as the first winner of the new Eat Out Irna van Zyl Food Media Icon Award. She has been awarded in recognition of her significant contribution to food content creation and coverage in South Africa.
All the winners celebrated at the 2019 Eat Out Mercedes-Benz Restaurant Awards are:
Eat Out S.Pellegrino & Acqua Panna Chef of the Year: Bertus Basson
Eat Out Restaurant of the Year: La Colombe
Top 10 Restaurants:
1. La Colombe
2. The Test Kitchen
3. The Restaurant at Waterkloof
4. Wolfgat
5. FYN
6. La Petite Colombe
7. Eike
8. Salsify
9. Restaurant Mosaic at The Orient
10. Chefs Warehouse at Beau Constantia
11. Camphors at Vergelegen
12. Overture
13. LivingRoom at Summerhill Estate
14. Chefs Warehouse at Maison
15. Greenhouse at The Cellars-Hohenort
16. Le Coin Français
17. La Tête
18. The Shortmarket Club
19. Tjing Tjing Momiji
20. Rust en Vrede


opkg install package- Remove a package:
opkg remove package- Update the list of available packages:
opkg update- Upgrade all the installed packages:
opkg upgrade- Upgrade one or more specific package(s):
opkg upgrade package(s)- Display information for a specific package:
opkg info package- List all the available packages:
opkg list


{% extends 'server_ui/templates/base.html' %} {% block title %}{{ settings.SITE_TITLE }}{% endblock %} {% block header %}
Helios offers verifiable online elections.
We believe democracy is important, whether it’s your book club, parent-teacher association, student government, workers’ union, or state. So we’ve made truly verifiable elections as easy as everything else on the Web.
Helios elections are:
More than 2,000,000 votes have been cast using Helios.
{% if create_p %} criar eleição {% endif %} {% else %}{{settings.WELCOME_MESSAGE|safe}}
{% if elections|length %}{% for election in elections %}
Selecione uma eleição para votar. Suas credenciais serão solicitadas após você preencher a cédula.
{% endif %} {% endif %}


| Personal Information | |
|---|---|
| Name: | {{ vm.coach.name | capitalize }} {{ vm.coach.surname | capitalize }} |
| Age | {{ vm.coach.age }} |
| Nationality |
| Professional Information | |
|---|---|
| Favourite Module | {{vm.coach.favouriteModule}} |
| Skill | {{vm.coach.skillAvg}} |
| Wage | {{vm.coach.wageReq | currency: ''}} € |


See: Description
| Interface | Description |
|---|---|
| MessagePreparedStatementSetter |
The callback to be used with the
JdbcMessageHandler
as an alternative to the SqlParameterSourceFactory . |
| SqlParameterSourceFactory |
Collaborator for JDBC adapters which allows creation of
instances of
SqlParameterSource for use in update operations. |
| Class | Description |
|---|---|
| BeanPropertySqlParameterSourceFactory |
A default implementation of
SqlParameterSourceFactory which creates an SqlParameterSource to
reference bean properties in its input. |
| ExpressionEvaluatingSqlParameterSourceFactory |
An implementation of
SqlParameterSourceFactory which creates
an SqlParameterSource that evaluates Spring EL expressions. |
| JdbcMessageHandler |
A message handler that executes an SQL update.
|
| JdbcOutboundGateway | |
| JdbcPollingChannelAdapter |
A polling channel adapter that creates messages from the payload returned by
executing a select query.
|
| StoredProcExecutor |
This class is used by all Stored Procedure (Stored Function) components and
provides the core functionality to execute those.
|
| StoredProcMessageHandler |
A message handler that executes Stored Procedures for update purposes.
|
| StoredProcOutboundGateway |
An
AbstractReplyProducingMessageHandler implementation for performing
RDBMS stored procedures which return results. |
| StoredProcPollingChannelAdapter |
A polling channel adapter that creates messages from the payload returned by
executing a stored procedure or Sql function.
|


Check the Groups of subscribers
Check Individual Subscribers
No Contacts could be found that match the search criteria
{{value}}
No Lists could be found that match the search criteria
{{vm.list.Name}}
{{vm.list.Contacts}}


The Chancellor of the Duchy of Lancaster and the Secretary of State for Northern Ireland have announced a £650m package of investment.
The government has launched a new campaign to help businesses and individuals prepare for the end of the transition period.
The UK’s approach to negotiations with the European Union.
Border controls for EU goods imported into Great Britain will be introduced in stages from the end of the transition period.


Pat's Cookie Stand
Home
About Us
Contact Us
Locations
Sales
Store Sales Forcast
Location
Min. Customers Per Hour
Max Customers Per Hour
Cookies Purchesed Per Hour
Submit Data


html2text(1) html2text(1) NAME html2text - an advanced HTML-to-text converter SYNOPSIS html2text -help html2text -version html2text [ -unparse | -check ] [ -debug-scanner ] [ -debug-parser ] [ -rcfile path ] [ -style ( compact | pretty ) ] [ -width width ] [ -o out‐ put-file ] [ -nobs ] [ -ascii | -utf8 ] [ -nometa ] [ input-url ... ] DESCRIPTION html2text reads HTML documents from the input-urls, formats each of them into a stream of plain text characters, and writes the result to standard output (or into output-file, if the -o command line option is used). If no input-urls are specified on the command line, html2text reads from standard input. A dash as the input-url is an alternate way to specify standard input. html2text understands all HTML 3.2 constructs, but can render only part of them due to the limitations of the text output format. However, the program attempts to provide good substitutes for the elements it cannot render. html2text parses HTML 4 input, too, but not always as successful as other HTML processors. It also accepts syntactically incorrect input, and attempts to interpret it "reasonably". The way html2text formats the HTML documents is controlled by formatting properties read from an RC file. html2text attempts to read $HOME/.html2textrc (or the file specified by the -rcfile command line option); if that file cannot be read, html2text attempts to read /etc/html2textrc. If no RC file can be read (or if the RC file does not override all formatting properties), then "reasonable" defaults are assumed. The RC file format is described in the html2textrc(5) manual page. Debian version of html2text also can do input and output recoding (see /usr/share/doc/html2text/README.Debian for more info). html2text tries to fetch encoding from HTML document. If encoding is not specified, you can use -ascii and -utf8 options. Output is converted to user's locale charset (LC_CTYPE). OPTIONS -nometa By default, Debian version of html2text use 'meta http-equiv' tag for input recoding. This option cancels this behavior. -ascii By default, when -nometa is supplied, html2text uses UTF-8 for the output. Specifying this option, plain ASCII is used instead. To find out how non-ASCII characters are rendered, refer to the file "ascii.substitutes". -utf8 By default, when -nometa is supplied, html2text uses ISO 8859-1 for the input. Specifying this option, UTF-8 is used instead (both for input and output). This option implies -nobs. -check This option is for diagnostic purposes: The HTML document is only parsed and not processed otherwise. In this mode of operation, html2text will report on parse errors and scan errors, which it does not in other modes of operation. Note that parse and scan errors are not fatal for html2text, but may cause mis-interpreta‐ tion of the HTML code and/or portions of the document being swal‐ lowed. -debug-parser Let html2text report on the tokens being shifted, rules being applied, etc., while scanning the HTML document. This option is for diagnostic purposes. -debug-scanner Let html2text report on each lexical token scanned, while scanning the HTML document. This option is for diagnostic purposes. -help Print command line summary and exit. -nobs By default, original html2text renders underlined letters with sequences like "underscore-backspace-character" and boldface let‐ ters like "character-backspace-character". Because of issues with UTF-8, Debian version of html2text doesn't produce backspaces, so this option really does nothing. -o output-file Write the output to output-file instead of standard output. A dash as the output-file is an alternate way to specify the standard output. -rcfile path Attempt to read the file specified in path as RC file. -style ( compact | pretty ) Style pretty changes some of the default values of the formatting parameters documented in html2textrc(5). To find out which and how the formatting parameter defaults are changed, check the file "pretty.style". If this option is omitted, style compact is assumed as default. -unparse This option is for diagnostic purposes: Instead of formatting the parsed document, generate HTML code, that is guaranteed to be syn‐ tactically correct. If html2text has problems parsing a syntacti‐ cally incorrect HTML document, this option may help you to under‐ stand what html2text thinks that the original HTML code means. -version Print program version and exit. -width width By default, html2text formats the HTML documents for a screen width of 79 characters. If redirecting the output into a file, or if your terminal has a width other than 80 characters, or if you just want to get an idea how html2text deals with large tables and different terminal widths, you may want to specify a different width. FILES /etc/html2textrc System wide parser configuration file. $HOME/.html2textrc Personal parser configuration file, overrides the system wide val‐ ues. CONFORMING TO HTML 3.2 (HTML 3.2 Reference Specification - http://www.w3.org/TR/REC- html32), RESTRICTIONS Debian version of html2text have no http support. Use html2text through pipes with curl or wget instead. See README.Debian for more information. html2text was written to convert HTML 3.2 documents. When using it with HTML 4 or even XHTML 1 documents, some constructs present only in these HTML versions might not be rendered. AUTHOR html2text was written up to version 1.2.2 by Arno Unkrig<EMAIL_ADDRESS>for GMRS Software GmbH, Unterschleissheim. Current maintainer and primary download location is: Martin Bayer<EMAIL_ADDRESS>http://www.mbayer.de/html2text/files.shtml This man page was modified for Debian by Eugene V. Lyubimkin<EMAIL_ADDRESS>⟨⟩ SEE ALSO html2textrc(5), less(1), more(1) 2008-09-20 html2text(1)


To facilitate the community in finding ways to contribute that match their experiences and skillsets, we have developed a comprehensive system for labelling issues and PRs. This is an introduction to this standard labelling scheme.
Labels consist of three fields, viz. name, description and color. Label names
should have a consistent format to aid both filtering within the github UI as
well as scanning visually through the list. The following format is the most
suited to this task (where ⎵
denotes a single space):
<emoji>⎵<category>:⎵<name>
Even label names with more than two words in the text should be separated by a
single space ⎵
.
An issue has many different attributes:
| Priority color chart | ||||
| Unfavourable | Negative | Neutral | Positive | Favourable |
| #b60205 | #ff9f1c | #ffcc00 | #cfda2c | #008672 |
The priority of an issue is based on its impact, derived from a combination of urgency and importance. This determines the importance of the issue when sprint planning or deciding which issues to tackle next.
| Status color chart | ||||
| Lighter | Light | Medium | Dark | Darker |
| #eeeeee | #cccccc | #999999 | #666666 | #333333 |
The status of the issue determines whether it is ready for work or not. Issues may not be ready to be worked on for a number of reasons and the maintainers must keep updating the labels as the situation evolves.
An issue, at the time of closing can have either the 🏁 status: ready for dev or the ⛔ status: discarded label based on whether it was closed with or without resolution, respectively.
The goal of an issue is the end-result achieved when the issue is resolved. This represents the impact of the issue on the scope of the software.
The aspect of an issue is the side of the project that the issue deals with. A single codebase can have multiple aspects to it and knowing which ones will be touched helps contributors find relevant issues.
The technical skills a person is required to possess to work on the issue. Skills are a special type of label that vary by repository. Issues may not have a skill tag if no special skills are required.
Issues with interaction labels do not entail any work to be done on the repository. They are for Q&A, RFCs and any other form of discussions. While both of these may appear to be too similar, triage permissions are granted to collaborators who might not have the answers to questions but by labelling them as such, they might draw a faster response from an experienced contributor who does. In the future this category might be rendered redundant by GitHub Discussions and will be removed.
The level of friendliness a particular issue is the valency of the issue towards contributions from the community. Some issues provide a great introduction while others require a little more familiarity with the codebase. These issues do not have the category prefix as two of them are special labels recognized by GitHub. The special ones don't have emojis either.
In the above categories, some are marked with an *
sign. Those are mandatory
categories and every issue is required to have at least one label from those
categories.
Only issues labelled with 🚦 status: awaiting triage are allowed to not have all mandatory labels (seeing as they have not been triaged yet).
Categories not marked with an *
are optional and it is upto the discretion of
the maintainers to have apply labels from those categories to issues.


Read what Matthew 16:24-26 says about this irony of gaining something by letting go. Paraphrase this passage in your own words below:
If we cling to anything but God, even if we cling to our own self-sufficiency or our own false identities, we will find ourselves sinking into a pit we’ve dug for ourselves. But if we let go and grab hold of God, we’ll discover that we’re closer to finding the peace, forgiveness, and transformation we long for.
The fact that God loves us and chooses us is no guarantee that we won’t struggle. But it is a guarantee that we won’t struggle alone.
Instead of leaving Jacob alone in his time of struggle, God wrestled with Him through one of the most difficult times in his life. He also renamed him Israel, indicating that God would continue to wrestle with him—and with the people who would follow in his family—when times were tough.
Read Isaiah 43:1-7. What does God promise in this passage about the difficult times in our lives?
Looking back at the tough times in your life, has there been a time when you knew without a doubt that God was with you? If so, describe how you knew at the time, or how you know, in retrospect, that God joined you in your struggle.
When we wrestle with the issues of life—even issues we’ve brought on ourselves, like Jacob—God will be right there wrestling with us, bringing us an outcome of blessing, even when we don’t deserve it. God will always be there to offer a new blessing, a new name, and a new chance to be declared His children.
 On one side, list the things you’ve been clinging to for security, safety, and significance.
On the other side, write your name and God’s name. On this side, write a prayer asking God to help you cling only to Him and nothing else for security.


A hub for aesthetic images
Bon Voyage
Wanderlust
What's Up ?
Buckle up buttercup
Vroom...
Up in the clouds!

Eunoia
Cycling away...
Citylights
Royal
Serendipity


Fork me on GitHub
Tiny Land Online
A Tiny Online RPG Game (v0.8!)
Draw yourself:
left click: draw / right click: erase
Enter your name:
Create Hero
Top players:
Players Online:
HP: 10/10
LVL 1
Gold: 0
atk: 0
def: 0


{% extends 'administration/base_admin.html' %} {% block content %} {% load filter %}
Another admin or user already deleted this need!
{{ need|get_short_text:160 }}
{{ need.text|linebreaksbr }}
If you delete This need, this data will be permanently lost.
Would you like to rethink your decision?


A mind-map helps us understand the concepts and the relationships between them.
Advantages:
Adaptable
Easy to add ideas later, at any time
essay plans
Help you focus on the links and relationships between ideas so you don't just have disconnected facts.


loading reports data for {{date}}...
| Total:{{count}} | Total Expenses:{{totalExpense | number:2}} | Total Income:{{totalIncome | number:2}} | Balance:{{totalIncome - totalExpense | number:2}} |
| {{payment.createdAt | date}} |
{{payment.date | date}} |
{{payment.description}}
{{payment.description}} {{payment.description}}
|
Form Collection
Booking Due
|
{{payment.amount}} |
{{payment.amount}} |
{{payment.attributes.submittedBy}} |
{{payment.status}}
|
| No Payments |
| {{payment.createdAt | date}} |
{{payment.date | date}} |
{{payment.description}}
{{payment.description}} {{payment.description}}
|
Form Collection
Booking Due
|
{{payment.amount}} |
{{payment.amount}} |
{{payment.attributes.submittedBy}} |
{{payment.status}}
|
Please wait while the payments data is loaded


Color and thicknesses are supported (styles are not):
1 px solid
3 px dotted
0 px dotted
solid thick
inset thick
thick groove red
yellow ridge 1cm
inset thick blue
thin solid red


Definition of Big Data
Volume: Data coming in from new sources as well as increased regulation in multiple areas means storing more data for longer periods of time. Variety: Unstructured and semi-structured data is becoming as strategic as traditional structured data and is
growing at faster rates. Velocity: Social media, RFID, machine data, etc. are needing to be ingested at speeds not even imagined a few years ago.
Estimated 1.5 terabytes of compressed data is produced daily.
With over 6 million paying subscribers, 1.5 billion playlists and more than 20 million songs, an estimated 1.5 terabytes of compressed data is produced daily. Terabytes of data is analyzed daily in order to get statistics about Spotify and our users,
learn about artist trends, target advertisements and do better music recommendations. The goal is to use data to make a better music service. Interesting Fact: The company used its collection of data last year to see whether the decisions
of the Recording Academy (the people who vote on the winners at the Grammy Awards) reflected the habits the public on the streaming service. Out of 8 categories, Spotify attempted to predict the winners by looking at its users’ data (listening
habits, subscribers to a playlist, popularity of an artist, etc.) The accuracy? 67%. That figure may seem low, but you should consider how many nominees had to be considered.
Twitter messages are 140 bytes each generating 8TB data per day.
A Boeing 737 will generate 240 terabytes of flight data during a single flight across the US
Facebook has around 50 PB warehouse and it’s constantly growing.
Why invest time and money into Big Data?
Source for following slides: http://www.researchmoz.us/big-data-market-business-case-market-analysis- and-forecasts-2014-2019-report.html
IDC is predicting the big data market is expected to grow about 32% a year to $23.8 billion in 2016
The market for analytics software is predicted to reach $51 billion by 2016
Mind Commerce estimates global spending on big data will grow 48% between 2014 and 2019
Big data revenue will reach $135 billion by the end of 2019
Structured data is any data that is captured in keys, records, attributes, and indexes in a standard DBMS.
Unstructured data is any data that is not managed by a standard database management system (DBMS). Such data comes in many formats, including text, document, image, video, and more.
Most Common Types of Data
Sentiment: The most commonly sighted source, analyzing language usage, text and computational iinguistics in an attempt to better analyze subjective information. Many companies are trying to leverage this data to provide sentiment trackers, identify influencers
etc. Clickstream: The trail a user leaves behind as he navigates your website. Analyze the trail to optimize website design. Sensor/Machine: These are everywhere: cars, health equipment, smartphones, etc. Nike put one in shoes. Someone also
put one in baby diapers! They call it ‘proactive maintenance’. Geographic: Location based data – a common use being location based targeting. This data has much more wider application in supply chain optimization across the manufacturing industry
allowing organizations to optimize routes, predict inventory levels, etc. Server logs: This one is not new to the IT world. You often lose precious trails and information when you simply roll over log files. Today, you should not have to lose
this data; you just save the data in Hadoop! Text: Text is everywhere. We all love to express ourselves - every blog, article, news site, ecommerce site you go these days, you will find people putting out their thoughts. And this is on top
of the already existing text sources like surveys and the Web content itself. How do you store, search and analyze all this text data to glean for key insights? Hadoop!
Big Data Technologies
Big Data Short Video
Top ten Big Data Insights at Rackspace
10 Minute Video
What is Big Data to Rackspace?
7 Minutes
Hadoop
Lots of things can be "Big Data". Today we will focus on Hadoop.
What is Hadoop?
The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models.
It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.
Hadoop is all about processing and storage. Hadoop is a software framework environment that provides a parallel processing environment on a distributed file system using commodity hardware.  This framework allows someone to build a Hadoop cluster that offers high performance super computer capability. WHITEBOARD four master nodes and 10 data nodes and explain how Hadoop does parallel
processing across data nodes on a highly available distributed file system.
Evolution of Hadoop
Traditional Systems vs. Hadoop
Traditional Systems vs. Hadoop Hadoop is not designed to replace existing relational databases or data warehouses. Relational databases are designed to manage transactions. They contain a lot of feature/functionality designed around managing transactions.
They are based upon schema-on-write. Organizations have spent years building Enterprise Data Warehouses (EDW) and reporting systems for their traditional data. The traditional EDWs are not going anywhere either. EDWs are also based on schema-on-write.
Apache Hadoop Framework
Hadoop Configuration
Name Nodes - is the centerpiece of a HDFS file system. It keeps a directory tree of all files in the file system, and tracks data location within the cluster. It does not store the data, it only keeps track of where it lives.
Gateway Node - Gateway Nodes are the interface between the cluster and outside network. They are also used to run client applications, and cluster administration services, as well as serving a staging area for data ingestion.
Data Nodes - These are the nodes where the data lives, and where the processing occurs. They are also referred to as “slave nodes”.
One of the benefits of Hadoop is that it is highly scalable, and it was designed to run on commodity hardware, as opposed to high-end, purpose built hardware.

Hadoop Cluster
Overview of a Hadoop Cluster Hadoop cluster consists of the following components: - NameNode: a master server that manages the namespace of HDFS. - DataNodes: slave servers that store blocks of data. - ResourceManager: the master server of the YARN processing
framework. - NodeManagers: slave servers of the YARN processing framework. - HBase components: HBase also has a master server and slave servers called RegionServers. Some of the components working in the background of a cluster include ZooKeeper,
Ambari, Ganglia, Nagios, JobHistory, HiveServer2, and WebHCat.
Cloud Big Data Architecture
Managed/Dedicated Big Data Architecture
Test your Knowledge
this is a transition slide
Is Hadoop a data store?
No. While you can technically store data in a Hadoop Cluster, it is really more of a data processing system, than a true data store.
Ask the group this question. If so someone says yes, ask the larger group if that's the correctn answer. When someone says no, ask the group why it isn't. The goal here is to build an open discussion.
Then what is a data store?
A data store is a data repository including a set of integrated objects. These objects are modeled using classes defined in database schemas. Examples of typical data stores, MySQL, PostgreSQL, Redis, MongoDB
Ask the group this question. The goal here is to build an open discussion.
What is data processing?
Think of Data Processing as the collection and manipulation of items of data to produce meaningful information. In the case of Hadoop, this means taking unstructured, unusable data, and processing it in a way that makes it useful or meaningful
Ask the group this question. The goal here is to build an open discussion.
this is a transition slide
YARN Architecture
generate a naming convention for the nodes
YARN Use Case
At one point Yahoo! had 40k+ nodes spanning multiple datacenters – 365PB+ of data
YARN provided a compute framework that allowed higher utilization of nodes
100% utilization (while being efficient) is always a good thing
Yahoo! was able to bring down an entire datacenter of about 10k Nodes
Source: Hortonworks
HDFS
Java-based distributed file system.
Stores large volumes of unstructured data and spans across large clusters of commodity servers.
Works closely with MapReduce.
Hadoop is flexible. It lets you use other filesystems like GlusterFS, and it has a number of plugins for a number of other data storage layers. MapR wrote its own distributed filesystem that uses an NFS interface and relies on local filesystems rather
than using Namenodes, and they claim some impressive performance numbers with their system. You can also plug it in to object stores like OpenStack Swift, as well as distributed databases like MongoDB and Cassandra. HDFS will be your primary
concern, but you should be aware that these other options exist.
HDFS Architecture
The primary data processing framework for big data on Hadoop is MapReduce, which is based on the paper published by Google in 2004. The basic idea is that you process data in parallel by passing the processing code along to the location where
the data lives (since the data is larger than the code in these cases). You map your function across all the data in parallel, and then reduce those results into a singular resultset. This is the predominant way to process data in Hadoop,
and for a long time was the only way to process data.
MapReduce
Framework for writing applications that process large amounts of structured and unstructured data.
Designed to run batch jobs that address every file in the system.
Splits a large data set into independent chunks and organizes them into key, value pairs for parallel processing.
The primary data processing framework for big data on Hadoop is MapReduce, which is based on the paper published by Google in 2004. The basic idea is that you process data in parallel by passing the processing code along to the location where
the data lives (since the data is larger than the code in these cases). You map your function across all the data in parallel, and then reduce those results into a singular resultset. This is the predominant way to process data in Hadoop,
and for a long time was the only way to process data.
MapReduce Architecture
Spark Data Processing
Alternative to the traditional batch map/reduce model that can be used for real-time stream data processing and fast interactive queries that finish within seconds.
Spark is ideal for in-memory data processing. It allows data scientists to implement fast, iterative algorithms for advanced analytics such as clustering and classification of datasets. Spark is a is an alternative to Tez in many respects, as it also
provides a way to process data using a DAG (directed acyclic graph), but it’s also an alternative to Pig as it can be run natively within Java, Scala, or Python programs to execute functions across the data in your cluster with ease. Spark
can offer performance up to 100 times faster than Hadoop MapReduce for certain applications.
MapReduc e writes results to disk.
Spark holds intermediate results in memory.
MapReduce supports map and reduce functions.
Spark supports more than just map and reduce functions.
Pig
Allows you to write complex MapReduce transformations using a simple scripting language called "Pig Latin". It's made of two components:
Pig Latin (the language) defines a set of transformations on a data set such as aggregate, join and sort.
Compiler to translate Pig Latin to MapReduce so it can be executed within Hadoop.
Pig is a Hadoop-based language developed by Yahoo. It is relatively easy to learn and is adept at very deep, very long data pipelines (a limitation of SQL.)<
Pig
Hive
Used to explore, structure and analyze large datasets stored in Hadoop's HDFS.
Provides an SQL-like language called HiveQL with schema on read and converts queries to map/reduce jobs.
Hive is the oldest and most popular projects that provide an interface to Hadoop. Allows users to write queries in a SQL-like language caled HiveQL, which are then converted to MapReduce. This allows SQL programmers with no MapReduce experience
to use the warehouse and makes it easier to integrate with business intelligence and visualization tools Great website for Hive: https://cwiki.apache.org/confluence/display/Hive/Home
Hive Architecture
This diagram shows the major components of Hive and its interactions with Hadoop. As shown in this diagram, the main components of Hive are: 1. UI - The user interface for users to submit queries and other operations to the system. Currently the
system has a command line interface and a web based GUI is being developed. 2. Driver - The component which receives the queries. This component implements the notion of session handles and provides execute and fetch APIs modeled on JDBC/ODBC
interfaces. 3. Compiler - The component that parses the query, does semantic analysis on the different qurey blocks and query expressions and eventually generates an execution plan with the help of the table and partition metadata looked up
from the metastore. 4. Metastore - The component that stores all the structure information of the various table and partitions in the warehouse including column and column type information, the serializers and deserializers necessary to read
and write data and the corresponding hdfs files where the data is stored. 5. Execution Engine - The component which executes the execution plan created by the compiler. The plan is a DAG of stages. The execution engine manages the dependencies
between these different stages of the plan and executes these stages on the appropriate system components.
Hive
Spark SQL
Part of the Spark computing framework.
Allows relational queries expressed in SQL, HiveQL, or Scala to be executed using Spark.
Used for real-time, in-memory, parallelized processing of Hadoop data.
Spark SQL
Spark SQL Use Cases
Stand Alone: Standalone deployment: With the standalone deployment one can statically allocate resources on all or a subset of machines in a Hadoop cluster and run Spark side by side with Hadoop MR. The user can then run arbitrary Spark jobs on her HDFS
data. Its simplicity makes this the deployment of choice for many Hadoop 1.x users. Spark Over Yarn: Hadoop Yarn deployment: Hadoop users who have already deployed or are planning to deploy Hadoop Yarn can simply run Spark on YARN without
any pre-installation or administrative access required. This allows users to easily integrate Spark in their Hadoop stack and take advantage of the full power of Spark, as well as of other components running on top of Spark. Spark In MapReduce
(SIMR): For the Hadoop users that are not running YARN yet, another option, in addition to the standalone deployment, is to use SIMR to launch Spark jobs inside MapReduce. With SIMR, users can start experimenting with Spark and use its shell
within a couple of minutes after downloading it! This tremendously lowers the barrier of deployment, and lets virtually everyone play with Spark.
TEZ
Extensible framework for building YARN based, high performance batch and interactive data processing applications.
It allows applications to span the scalability dimension from GB’s to PB’s of data and 10’s to 1000’s of nodes.
Apache Tez component library allows developers to use Tez to create Hadoop applications that integrate with YARN and perform well within mixed workload Hadoop clusters.
TEZ
Using Tez to create Hadoop applications that integrate with YARN
The Apache Tez component library allows developers to use Tez to create Hadoop applications that integrate with YARN and perform well within mixed workload Hadoop clusters. website with more information: http://hortonworks.com/hadoop/tez/
TEZ
By allowing projects like Apache Hive and Apache Pig to run a complex DAG of tasks, Tez can be used to process data, that earlier took multiple MR jobs, now in a single Tez job as shown above.
Apache Tez component library allows developers to use Tez to create Hadoop applications that integrate with YARN and perform well within mixed workload Hadoop clusters.
HBase
Column oriented database that allows reading and writing of data to HDFS on a real-time basis.
Designed to run on a cluster of dozens to possibly thousands or more servers.
Modeled after Google's Bigtable
EBay and Facebook use HBase heavily.
The active link on this slide is for a 1 minute and 42 second video that covers HBase. It's slides with music in the background.
When to use HBase?
When you need random, realtime read/write access to your Big Data.
When hosting very large tables -- billions of rows X millions of columns -- atop clusters of commodity hardware.
HBase 101 Architecture
Data streaming provides the ability to manipulate data as it enters or exits the system. Think of it as the big data version of an RSS reader, except you can programmatically manipulate the feed as it’s generated.
Storm
Distributed real-time computation system for processing fast, large streams of data.
With Storm and MapReduce running together in Hadoop on YARN, a Hadoop cluster can efficiently process a full range of workloads from real-time to interactive to batch.
Storm Characteristics
Storm
Enterprises use Storm to prevent certain outcomes or to optimize their objectives. Here are some “prevent” and “optimize” use cases.
Storm
How Storm Works A storm cluster has three sets of nodes: Nimbus node (master node, similar to the Hadoop JobTracker): Uploads computations for execution Distributes code across the cluster Launches workers across the cluster Monitors computation and reallocates
workers as needed ZooKeeper nodes – coordinates the Storm cluster Supervisor nodes – communicates with Nimbus through Zookeeper, starts and stops workers according to signals from Nimbus
Spark Streaming
Part of the Spark computing framework.
Easy and fast real-time stream processing.
Provides a high-level abstraction called discretized stream or DStream, which represents a continuous stream of data.
Supports higher-level tools including Spark SQL for SQL and structured data processing, MLlib for machine learning, GraphX for graph processing, and Spark Streaming.
Spark Characteristics
Spark
Data can be ingested from many sources like Kafka, Flume, Twitter, ZeroMQ or plain old TCP sockets and be processed using complex algorithms expressed with high-level functions like map, reduce, join and window. Processed data can be pushed out
to filesystems, databases, and live dashboards. In fact, you can apply Spark’s in-built machine learning algorithms, and graph processing algorithms on data streams.
Spark
Internally, it works as follows. Spark Streaming receives live input data streams and divides the data into batches, which are then processed by the Spark engine to generate the final stream of results in batches.
this is a transition slide
Sqoop
Command-line interface application used to transfer data to and from Hadoop to any RDMS.
Sqoop: Importing to Hive
Flume
Application that collects, aggregates, and moves large amounts of streaming data into the Hadoop Distributed File System (HDFS).
Flume was the stream of tweets collected in Iron man 3.
Flume
Flume
Agents consist of three pluggable components: sources, sinks, and channels. An agent must have at least one of each in order to run.
Sources Flume sources listen for and consume events. Events can range from newline-terminated strings in stdout to HTTP POSTs and RPC calls — it all depends on what sources the agent is configured to use. Flume agents may have more than one source, but
must have at least one. Sources require a name and a type; the type then dictates additional configuration parameters.On consuming an event, Flume sources write the event to a channel. Importantly, sources write to their channels as transactions.
By dealing in events and transactions, Flume agents maintain end-to-end flow reliability. Events are not dropped inside a Flume agent unless the channel is explicitly allowed to discard them due to a full queue. Channels Channels are the mechanism
by which Flume agents transfer events from their sources to their sinks. Events written to the channel by a source are not removed from the channel until a sink removes that event in a transaction. This allows Flume sinks to retry writes in
the event of a failure in the external repository (such as HDFS or an outgoing network connection). For example, if the network between a Flume agent and a Hadoop cluster goes down, the channel will keep all events queued until the sink can
correctly write to the cluster and close its transactions with the channel. Channels are typically of two types: in-memory queues and durable disk-backed queues. In-memory channels provide high throughput but no recovery if an agent fails.
File or database-backed channels, on the other hand, are durable. They support full recovery and event replay in the case of agent failure. Sinks Sinks provide Flume agents pluggable output capability — if you need to write to a new type storage,
just write a Java class that implements the necessary classes. Like sources, sinks correspond to a type of output: writes to HDFS or HBase, remote procedure calls to other agents, or any number of other external repositories. Sinks remove
events from the channel in transactions and write them to output. Transactions close when the event is successfully written, ensuring that all events are committed to their final destination.
Ambari
Operational framework for provisioning, managing and monitoring Apache Hadoop clusters.
Ambari is a 100% Apache open source operations framework for provisioning, managing, and monitoring Hadoop clusters. It provides these features through a web frontend and an extensive REST API. With Ambari, clusters can be built from ground up
on clean operating system instances. It will do the job of propagating binaries, configuring services, launching them, and monitoring them to all the hosts in a cluster.
Ambari Dashboard
Zookeeper
Provides configuration service, synchronization service, and naming registry for software in the Hadoop ecosystem.
Distributed applications use ZooKeeper to store and mediate updates to important configuration information.
Zookeeper Benefits
Zookeeper
Clicl on the title's hyperliunk to learn how zookeeper works in the diagram
Solr
Searches data stored in HDFS in Hadoop.
Powers the search and navigation features of many of the world’s largest Internet sites, enabling powerful full-text search and near real-time indexing.
Solr Features
Near real-time indexing
Advanced full-text search
Comprehensive HTML administration interfaces
Flexible and adaptable, with XML configuration
Server statistics exposed over JMX for monitoring
Standards-based open interfaces like XML, JSON and HTTP
Linearly scalable, auto index replication, auto failover and recovery
Hue
An open-source Web interface that supports Apache Hadoop and its ecosystem.
Hue is one of the many out of the box methods that HDFS provides for clients to interact with the file system.
DEMO
Hue
Oozie
Workflow scheduler system to manage Apache Hadoop jobs.
There are two basic types of Oozie jobs:
Serves the same purpose as Tez, but predates Tez and runs on top of other Hadoop constructs rather than below them.
Oozie Architecture
Falcon
Framework that simplifies data management by allowing users to easily configure and manage data migration, disaster recovery and data retention workflows.
Data management framework for simplifying data lifecycle management and processing pipelines on Apache Hadoop®. It enables users to orchestrate data motion, pipeline processing, disaster recovery, and data retention workflows. Falcon provides the key
services data processing applications need. Falcon manages workflow and replication. Falcon’s goal is to simplify data management on Hadoop. It achieves this by providing important data lifecycle management services that any Hadoop application
can rely on. Instead of hard-coding complex data lifecycle capabilities, apps can now rely on a proven, well-tested and extremely scalable data management system built specifically for the unique capabilities that Hadoop offers. Falcon also
supports multi-cluster failover.
Falcon Architecture
KNOX
Provides a single access point for Hadoop services in a cluster.
Knox runs as a server (or cluster of servers) that serve one or more Hadoop clusters.
Knox is intended to provide permiter security. It aims to provide a single point of entry into a Hadoop cluster for a user to access different services such as HDFS, YARN, Hive, Oozie. Knox can be installed in HDP 2 as an add-on. A user authenticates
once with the Knox service via Kerberos, while Knox itself handles serving requests for that user inside the cluster.
Knox Advantages
Knox Architecture
Let's take another look at the ecosystem
Find a use case for your group's assigned component
1. Form 2-3 person groups 2. Assgn each group to one of the columns...Data Processing | SQL Engines | NoSQL Engines | Data Ingest/Management | Stream Processing 3. Have the groups research a Hadoop use case for one or both of the Hadoop components in
their assigned column 4. Have groups report their findings
Hadoop Ecosystem
Hadoop Use Cases
Lots of things can be "Big Data". Today we will focus on Hadoop.
five major types of Hadoop data that organizations are using to increase business value, including everything from decreasing costs, optimizing security processes, and building value. Sentiment: The most commonly sighted source, analyzing language usage,
text and computational iinguistics in an attempt to better analyze subjective information. Many companies are trying to leverage this data to provide sentiment trackers, identify influencers etc. Clickstream: The trail a user leaves behind
as he navigates your website. Analyze the trail to optimize website design. Sensor/Machine: These are everywhere: cars, health equipment, smartphones, etc. Nike put one in shoes. Someone also put one in baby diapers! They call it ‘proactive
maintenance’. Geographic: Location based data – a common use being location based targeting. This data has much more wider application in supply chain optimization across the manufacturing industry allowing organizations to optimize routes,
predict inventory levels, etc. Server logs: This one is not new to the IT world. You often lose precious trails and information when you simply roll over log files. Today, you should not have to lose this data; you just save the data in Hadoop!
Text: Text is everywhere. We all love to express ourselves - every blog, article, news site, ecommerce site you go these days, you will find people putting out their thoughts. And this is on top of the already existing text sources like surveys
and the Web content itself. How do you store, search and analyze all this text data to glean for key insights? Hadoop!
Sentiment Data
Data collected from social media platforms ranging from Twitter, to Facebook, to blogs, to a never-ending array websites with comment sections and other ways to socially interact on the Internet.
While clickstreams will give a sort of Boolean understanding of customers and their actions, this is the era of social media where organizations can get a more subjective appreciation of what their customers are thinking about them (and their competitors),
and take actions to respond to these sentiments. From Twitter, to Facebook, to blogs, to a never-ending array websites with comment sections and other ways to socially interact on the Internet, there is the potential to be a massive amount
of sentiment information available about any given company. That unstructured data creates problems for traditional databases; however, unstructured data is where Hadoop shines. An organization can collect all of these data streams and track
how their customers and prospects feel about its products, the company itself, issues important to the company, competitors and more. Correlating information about prevailing sentiments and their locations, a company can custom target their
marketing and ad campaigns to either capitalize on or combat the sentiments.
Sentiment Data
This use case is for sentimental and geographic data Hadoop was used to track the volume of tweets around the movie’s launch. Three clear spikes in global tweet volume were identified, including the Friday premier, the Saturday matinee time, and the Saturday
evening showing. Correlating location data with sentiment analysis, the studio was able to see that in Ireland, over half the tweets expressed positive sentiments about the movie, where in Mexico, negative sentiments about the movie outweighed
the positive ones. These sentiments were then tracked during the launch for real time marketing activities (i.e. increasing spends and extending campaigns in a particular market ), as well as planning future movie launches.
Sensor/Machine Data
Data collected from sensors that monitor and track very specific things, such as temperature, speed, location
Sensor data is among the fastest growing data types, with data collectors being put on everything under the sun. These sensors monitor and track very specific things, such as temperature, speed, location – if it can be tracked, there’s a sensor to track
it. People are carrying sensors on them regularly (smart phones), and more come online every day. Hadoop is a very attractive store for sensor data due to the ability to dump so much of the juice into the framework and then use analysis tools
to extract correlations that give insights on operations, and how things change when conditions change. Sensor data promises to be huge, as the desire to monitor the moment-to-moment status of things increases as businesses look for ways to
increase efficiency and cut costs in their operations. With every second, another data point needs to be stored, creating a challenge for existing traditional database paradigms. The relative cheapness of Hadoop makes it a very attractive
candidate for sensor data storage.
Sensor Data
HVAC
Used Hadoop to monitor the heating ventilation and air conditioning also known as HVAC in 20 large buildings around the world. Sensors and each HVAC system wirelessly transmitted temperature data that was combined with other data to maintain comfortable
indoor environments and minimize heating and cooling expense. They loaded, refined and visualized the sensor data. They loaded sensor data on the target and actual temperatures in the buildings. It flowed into the Hadoop distributed filesystem
known as HDFS via apache flume. They also used apache sqoop to import structured data on the HVAC units into HDFS. Think of this as a data leg that contains years of data of their heating and cooling systems.
Geographic Data
Geolocation data gives organizations the ability to track every moving aspect of their business, be they objects or individuals.
Geolocation data gives organizations the ability to track every moving aspect of their business, be they objects or individuals. In the age of smart phones, companies now have the ability to track their customers in ways that have previously been unimaginable.
Geographic Data
Geolocation A trucking company used Hadoop to analyze geolocation data, reduce fuel cost, and improve driver safety. Geolocation identifies the location of an object or individual at any point in time. This data might take the form of a cortinate or actual
street address. Geolocation data is used to locate people or assets. For people enterprises can learn where and when customer contegrate. For assets, transportation and logistics companies can better maintain their vehicles and control risks.
This company placed sensors on 100 long haul trucks. The sensors communicate the position and speed of each vehicle. They also sense unsafe events such as speeding or swerving. The company loads, refines and visualizes this geolocation data.
Geolocation data from the trucks along with other types of data is loaded into HDFS. Apache Flume streams geolocation data into HDFS and use apache sqoop to import structured truck data from a database.
Clickstream Data
Stream of clicks that a user takes as they path through a website.
Path Optimization
Basket Analysis
Next Product to Buy Analysis
Allocation of Website Resources
Granular Customer Segmentation
Path Optimization – Path optimization aims at reducing bounce rates and improving conversions. Basket Analysis – This aims at understanding aggregate customer purchasing behavior by examining such things as customer interests, and paths to purchase –
when customers bought Product X, what common paths did they take to get there. Next Product to Buy Analysis – Related to basket analysis, this type of analysis looks at correlation in purchases, and what can be offered next to help provide
more immediate value to the customer, and increase the likelihood of another sale. Allocation of Website Resources – Having clickstream data on hand, a company will know what their hottest and coldest paths on the site are and can assign development
resources accordingly, optimizing resource allocation. Granular Customer Segmentation – With clickstream and correlated user data, a company can discover and gain insight on how particular segments and micro-segments of customers are using
the site, and how to best cater to them. Clickstream analysis is all about measuring users and their behavior. With the data collected, the hot and cold spots on a website can be located and actions taken to ensure that users are seeing the
most valuable offers.
Clickstream Data
Clickstream: Path Optimization. A company wanted to change their website to reduce bounce rates and improve conversion. They used Hadoop to transform the sites data into three steps: Load, refine, visualize the click stream. First they loaded the raw
web logs with customer and product data into HDFS. Then they moved clickstream files into HDFS. Hadoop allowed them to mount the HDFS file system directly to their computer so the data could be loaded with a simple drag and drop. The clickstream
file contained 5 days of clickstream data, about four million (4,000,000) rows. After sorting the bounce pages they could see that there were four top candidates for improvement they used hadoop to drive this insight from a huge clickstream
web log combined with customer and product records all in the data leg. This was done in just a matter of minutes.
Server Log Data
Server log data is for security analysis on a network.
Some estimates say that global internet traffic will reach 1 zettabyte (the equivalent of 1 billion terabytes) by 2015. All of this data, as well as all assorted flavors of network traffic, winds up in server logs and is considered “exhaust data.” One
of the chief uses for server log data is for security analysis on a network. Admins are able to load their server logs into Hadoop using applications like Apache Flume, building a repository that they can use for analysis in order to identify
and repair vulnerabilities. Server log data can be used for an array of purposes to give organizations insights on everything from network usage, security threats, and compliance. Hadoop will be a central player in staging and analyzing this
type of data for some time to come.
Server Log Data
A System Administrator's system looks good when he went to lunch. When he returned he noticed a ten fold spike in the number of support tickets. And then his phone begins to ring off the hook. Members of his enterprise were unable to login to the VPN.
He captured and stored the server logs into the HDPS. This allowed him to see the logs from the VPN server flowing through Flume to HDFS. He noticed a distributed denial of service or DDOS attack. He could tell when the network traffic increased
and could see that most of the traffic came from unauthorized machines. This helped him determine that this was a DDOS attack. He was also able to see that the traffic was coming from several different companies where his company doesn't have
a presence. He used this data to update the firewall, deny requests from unauthorized IP addresses, and in a matter of minutes VPN access returns to his coworkers all over the globe.
Unstructured Video
Data from non-text-based things; that includes both images and also video.
If you were to store a bunch of images in a database, what you would really see in a database is they're just blobs. You can take a database, you can store it, but you’ve basically turned a database into just storage. You can’t really express
a SQL function that does anything with those images meaningfully. In Hadoop, yes, you can store it, but you can actually process it and analyze it. This is one of the nice bits of flexibility that you get with the MapReduce framework: When
you do MapReduce in Hadoop, what you do for your Reduce function can be quite varied. That Reduce step could be things like study an image.
Unstructured Video
Sky Box Imaging Use Case Sky Box Imaging is a satellite company. They launch a bunch of commodities satellites and take all kinds of overhead images of things going on in the world and then run that through imaging processing, which is scaled out using
MapReduce. They're basically grabbing lots of pictures of the world and they're using Hadoop to refine that and put it into valuable information. Then they're selling that information as a service to large corporate customers. For example,
is in the oil and gas industry, if they want to actually count out how many refining pads there are in a given site in the world is because you're trying to understand the aggregate capacity of your industry. That’s something that’s actually
possible to do. If you want to count how many cars are in a retailer’s parking lot, right, that’s an image-processing thing you can also do. So these are the types of things they're doing. A more complex but interesting use case... Astronomy
department at the University of Washington processing astronomical images in Hadoop. Astronomy in the Cloud: Using MapReduce for Image Coaddition!!! Astronomical surveys of the sky will generate tens of terabytes of images and detect hundreds
of millions of sources every night. link to whitepaper: http://arxiv.org/pdf/1010.1015v1.pdf Link to slides: http://www.slideshare.net/ydn/8-image-stackinghadoopsummit2010
Unstructured Text
Data collected from free-flowing text sources such as documents, emails, and tweets.
There is unstructured data, then there is unstructured text/content. There is a difference. Unstructured text/content refers to documents, emails, and other objects that are made up of free-flowing text. The contents of a Tweet, for example, are a type
of unstructured text (while Tweet metadata – when it was posted and by who – is unstructured data). Unstructured text lacks just about any form or structure we commonly associate with traditional corporate data. The text can be in any language,
follow (or not follow) accepted grammatical rules, and/or mix words and numbers. But the big difference between unstructured data and unstructured text is not what it is but what you do with it. Unstructured text can be mined for the same
type of patterns, but it also holds a treasure trove of insight that neither structured nor unstructured data offer: human sentiment. Unlike most unstructured data, humans, not machines, generate unstructured text. Text written in Tweets,
project proposals, and personal emails can reveal what people are thinking and feeling. Three steps for processing data from free flowing text: 1. Extraction: each document is split up into its individual words, and each word is counted by
the map function 2. Summarization: the framework groups all identical words with the same key and feeds them to the same call to reduce 3. Analysis: for a given word, the function sums all of its input values to find the total appearances
of that word
Unstructured Text
The process of legal reasoning and argument is largely based on information extracted from a variety of documents. Lawyers are paid large hourly sums to analyze documents to build their cases. Hadoop can help make this manual review process more efficient.
Firms can store the documentation in Hadoop, and then analyze that text en masse using processes like natural language processing or text mining. This allows legal researchers to search documents for important phrases and then use Hadoop ecosystem
solutions to analyze relationships between those and other phrases. This preliminary analysis optimizes the lawyer’s time reviewing the text, so she can read the parts that really matter. Source: http://hortonworks.com/wp-content/uploads/2014/05/Hortonworks.BusinessValueofHadoop.v1.0.pdf
Analytics Compute Grid
Analytics Compute Grid is a data analysis project
Rackspace uses Hadoop along with other big data technologies to understand customer patterns and identify trends.
This allows us to be better at delivering fanatical support to our customers using multiple data sources and systems.
Rackspace Analytics Compute Grid
The old model had a bunch of peace milled parts trying to integrate
Big Data: After Re-Architecting for Private Cloud
We used Openstack and new technologies like Hadoop and Cassandra to aggregate and make sense of all the data sources. This gives us a single view of customer info.
Popular Companies Leveraging Hadoop
Data Refinement... Distilling large qualities of structured and unstructured data for use in a traditional DW, e.g.: "sessionizatiuon" of weblogs
Data Exploration across unstructured content on millions of customer satisfaction surveys to determine sentiment.
Application Enrichment provides recommendations and personalized experience to website for each unique visitor.
Financial Services
Telecom
Retail
Manufacturing
Healthcare
Oil and Gas
Pharmaceutical


Sudoku Class Reference
Inherits Example.
Inherited by SudokuCpltSet [virtual]
, SudokuInt [virtual]
, and SudokuSet [virtual]
.
Detailed Description
Base class for Sudoku puzzles.Definition at line 65 of file sudoku.cc.
Sudoku specifications | |
| Each specification gives the initial positions that are filled in, with blank squares represented as zeroes. | |
| const char * | examples [] |
| The specifications. | |
| const unsigned int | n_examples = sizeof(examples)/sizeof(char*) |
| The number of instances. | |
| int | example_size (const char *s) |
| The size of an instance. | |
| int | sudokuField (const char *s, int n, int i, int j) |
| Return value at position (i,j) in the example s of size n. | |
Public Types | |
| enum | { MODEL_INT, MODEL_SET, MODEL_MIXED, MODEL_CPLT } |
| Model variants. More... | |
Public Member Functions | |
| Sudoku (const SizeOptions &opt) | |
| Constructor. | |
| Sudoku (bool share, Sudoku &s) | |
| Constructor for cloning s. | |
Protected Attributes | |
| const int | n |
| The size of the problem. |
Member Enumeration Documentation
| anonymous enum |
Constructor & Destructor Documentation
| Sudoku::Sudoku | ( | const SizeOptions & | opt | ) | [inline] |
| Sudoku::Sudoku | ( | bool | share, | |
| Sudoku & | s | |||
| ) | [inline] |
Friends And Related Function Documentation
const unsigned int n_examples = sizeof(examples)/sizeof(char*) [related] |
| int example_size | ( | const char * | s | ) | [related] |
| int sudokuField | ( | const char * | s, | |
| int | n, | |||
| int | i, | |||
| int | j | |||
| ) | [related] |
Member Data Documentation
The documentation for this class was generated from the following file:
- examples/sudoku.cc (Revision: 6034)