Hadoop
Hadoop :Meeting Big data Challenge
Apache Hadoop meets the challenges of Big Data by simplifying the implementation of data-intensive, highly parallel distributed applications.
Hadoop is different from previous distributed approaches in the following ways:
- Data is distributed in advance.
- Data is replicated throughout a cluster of computers for reliability and availability.
- Data processing tries to occur where the data is stored, thus eliminating bandwidth bottlenecks.
Hadoop provides a powerful mechanism for data analytics, which consists of the following:
- Vast amount of storage
- Distributed processing with fast data access
- Reliability, failover, and scalability
Data Science in the Business World
The capability of Hadoop to store and process huge amounts of data is frequently associated with “data science” .Although the term was introduced by Peter Naur in the 1960s, it did not get wide acceptance until recently.
Business analysts study patterns in existing business operations to improve them. The goal of data science is to extract meaning from data. The work of data scientists is based on math, statistical analysis, pattern recognition, machine learning,high-performance computing,data warehousing, and much more. They analyze information to look for trends, statistics, and new business possibilities based on collected information.
Businesses are using Hadoop for solving business problems, with a few notable examples:
Enhancing fraud detection for banks and credit card companies — Companies are utilizing Hadoop to detect transaction fraud. By providing analytics on large clusters of commodity hardware, banks are using Hadoop, applying analytic models to a full set of transactions for their clients, and providing near-real-time fraud-in-progress detection.
Social media marketing analysis — Companies are currently using Hadoop for brand management, marketing campaigns, and brand protection. By monitoring, collecting, and aggregating data from various Internet sources such as blogs, boards, news feeds, tweets, and social media, companies are using Hadoop to extract and aggregate information about their products, services, and competitors, discovering patterns and revealing upcoming trends important for understanding their business.
Shopping pattern analysis for retail product placement — Businesses in the retail industry are using Hadoop to determine products most appropriate to sell in a particular store based on the store’s location and the shopping patterns of the population around it.
Traffic pattern recognition for urban development — Urban development often relies on traffic patterns to determine requirements for road network expansion. By monitoring traffic during different times of the day and discovering patterns, urban developers can determine traffic bottlenecks, which allow them to decide whether additional streets/street lanes are required to avoid traffic congestions during peak hours.
Content optimization and engagement — Companies are focusing on optimizing content for rendering on different devices supporting different content formats. Many media companies require that a large amount of content be processed in different formats. Also, content engagement models must be mapped for feedback and enhancements.
Network analytics and mediation — Real-time analytics on a large amount of data generated in the form of usage transaction data, network performance data, cell-site information, device-level data, and other forms of back office data is allowing companies to reduce operational expenses, and enhance the user experience on networks.
Large data transformation — The New York Times needed to generate PDF files for 11 million articles (every article from 1851 to 1980) in the form of images scanned from the original paper. Using Hadoop, the newspaper was able to convert 4 TB of scanned articles to 1.5 TB of PDF documents in 24 hours.
THE HADOOP ECOSYSTEM
Hadoop should be classified as an ecosystem comprised of many components that range from data storage, to data integration, to data processing, to specialized tools for data analysts.
Hadoop’s ecosystem consists of the following:
HDFS — A foundational component of the Hadoop ecosystem is the Hadoop Distributed File System (HDFS). HDFS is the mechanism by which a large amount of data can be distributed over a cluster of computers, and data is written once, but read many times for analytics. It provides the foundation for other tools, such as HBase.
MapReduce — Hadoop’s main execution framework is MapReduce, a programming model for distributed, parallel data processing, breaking jobs into mapping phases and reduce phases (thus the name). Developers write MapReduce jobs for Hadoop, using data stored in HDFS for fast data access. Because of the nature of how MapReduce works, Hadoop brings the processing to the data in a parallel fashion, resulting in fast implementation.
HBase — A column-oriented NoSQL database built on top of HDFS, HBase is used for fast read/write access to large amounts of data. HBase uses Zookeeper for its management to ensure that all of its components are up and running.
Zookeeper — Zookeeper is Hadoop’s distributed coordination service. Designed to run over a cluster of machines, it is a highly available service used for the management of Hadoop operations, and many components of Hadoop depend on it.
Oozie — A scalable workflow system, Oozie is integrated into the Hadoop stack, and is used to coordinate execution of multiple MapReduce jobs. It is capable of managing a significant amount of complexity, basing execution on external events that include timing and presence of required data.
Pig — An abstraction over the complexity of MapReduce programming, the Pig platform includes an execution environment and a scripting language (Pig Latin) used to analyze Hadoop data sets. Its compiler translates Pig Latin into sequences of MapReduce programs.
Hive — An SQL-like, high-level language used to run queries on data stored in Hadoop, Hive enables developers not familiar with MapReduce to write data queries that are translated into MapReduce jobs in Hadoop. Like Pig, Hive was developed as an abstraction layer, but geared more toward database analysts more familiar with SQL than Java programming.
The Hadoop ecosystem also contains several frameworks for integration with the rest of the enterprise:
Sqoop — is a connectivity tool for moving data between relational databases and data warehouses and Hadoop. Sqoop leverages database to describe the schema for the imported/exported data and MapReduce for parallelization operation and fault tolerance.
Flume — is a distributed, reliable, and highly available service for efficiently collecting, aggregating, and moving large amounts of data from individual machines to HDFS. It is based on a simple and flexible architecture, and provides a streaming of data flows. It leverages a simple extensible data model, allowing you to move data from multiple machines within an enterprise into Hadoop.
Hadoop’s ecosystem is growing to provide newer capabilities and components, such as the following:
Whirr — This is a set of libraries that allows users to easily spin-up Hadoop clusters on top of Amazon EC2, Rackspace, or any virtual infrastructure.
Mahout — This is a machine-learning and data-mining library that provides MapReduce implementations for popular algorithms used for clustering, regression testing, and statistical modeling.
BigTop — This is a formal process and framework for packaging and interoperability testing of Hadoop’s sub-projects and related components.
Ambari — This is a project aimed at simplifying Hadoop management by providing support for provisioning, managing, and monitoring Hadoop clusters.
Single-node installation
The goal of the document is to get Hadoop version 2.0.3-alpha up and running on a single node cluster.
Introduction
Hadoop is a framework written in Java for running applications on large clusters of commodity hardware and incorporates features similar to those of the Google File System (GFS) and of the MapReduce computing paradigm.
The tutorial has been tested with the following software versions:
- Ubuntu Linux 12.04 (You can download Ubuntu 12.04 from here)
- JDK 1.7 (Will also work with JDK 1.6)
Apache Hadoop has undergone a complete overhaul with vrsion 2, (MRv2) also called as YARN(Yet Another Negotiator) YARN architecture splits the function of JobTracker into Resource Manager and per Application Master. NodeManager is similar to the Task tracker, with the task being abstracted as container. Hadoop 2.0 also comes with HDFS federation and High Avialble for namenode.
The tutorial has been tested with the following software versions:
- Ubuntu Linux 12.04 (download Ubuntu)
- JDK 1.7 ( download JDK). The same tutorial works on JDK1.6 as well.
Prerequisites
A system with 2 GB RAM and Ubuntu 12.04 up and running.
Installation
Create Dedicated User
Create a user ‘gpuser’ and group ‘gp’. We will be running Hadoop with this userid
$sudo addgroup gp $sudo adduser --ingroup gp gpuser
Install JDK
Ignore this step if JDK is already installed. If not, Download JDK and install.
Login with gpuser to proceed with further installation steps.
Set JAVA_HOME by adding the following lines at the end of .bashrc file located in gpuser home directory /home/gpuser
export JAVA_HOME=$HOME/java/jdk1.7.0_17 export PATH=$PATH:$JAVA_HOME/bin
Verify the Java installation with the following command:
gpuser@ubuntu:~$source ~/.bashrc gpuser@ubuntu:~$java -version
You should see java version 1.7.* printed on the console.
Download and Extract Hadoop
Download Hadoop version 2.0.3
Extract hadoop at the home directory using the following command.
gpuser@ubuntu:~$ tar -xvzf hadoop-2.0.3-alpha.tar.gz
Setup SSH
Hadoop requires SSH access and manage its nodes running hadoop. We will configure SSH to connect to localhost without a password
Install SSH
gpuser@ubuntu:~$sudo apt-get install ssh gpuser@ubuntu:~$sudo apt-get install rsync
Generate public/private RSA key pairs using the following command.
gpuser@ubuntu:~$ ssh-keygen -t rsa -P "" Generating public/private rsa key pair. Enter file in which to save the key (/home/gpuser/.ssh/id_rsa): Created directory '/home/gpuser/.ssh'. Your identification has been saved in /home/gpuser/.ssh/id_rsa. Your public key has been saved in /home/gpuser/.ssh/id_rsa.pub. The key fingerprint is: 3c:f7:ca:f2:49:a7:05:ee:90:66:05:0f:7b:a9:63:fc gpuser@ubuntu The key's randomart image is: +--[ RSA 2048]----+ | | | | | o | | . = . | | S B | | . O o | | X o + | | +.B * | | oE | +-----------------+
Press Enter to save the key in the default location.
Enable ssh to the local machine using the following command.
gpuser@ubuntu:~$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
- Finally test the ssh setup by connecting to the local machine with the gpuser user.
gpuser@ubuntu:~$ ssh localhost The authenticity of host 'localhost (<IP_ADDRESS>)' can't be established. ECDSA key fingerprint is ca:96:1c:5a:38:f8:9f:99:45:d4:57:82:3c:1b:64:b7. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts. The programs included with the Ubuntu system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Welcome to Ubuntu 12.10 (GNU/Linux 3.5.0-17-generic i686) * Documentation: https://help.ubuntu.com/
Configure Hadoop
Hadoop requires the following environment variables to be set correctly
- HADOOP environement variables
Open .bashrc file in the home folder and add the following lines at the end
export HADOOP_HOME=$HOME/hadoop-2.0.3-alpha export HADOOP_MAPRED_HOME=$HOME/hadoop-2.0.3-alpha export HADOOP_COMMON_HOME=$HOME/hadoop-2.0.3-alpha export HADOOP_HDFS_HOME=$HOME/hadoop-2.0.3-alpha export YARN_HOME=$HOME/hadoop-2.0.3-alpha export HADOOP_CONF_DIR=$HOME/hadoop-2.0.3-alpha/etc/hadoop export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin
- Source the variables using the following command
gpuser@ubuntu$source ~/.bashrc
- Update Configuration Files
Hadoop configration files at located at $HADOOP_HOME/etc/hadoop. Update the configuration files with the following entries respectively.
yarn-site.xml
<configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce.shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> </configuration>
Note:In case of hadoop 2.2.x installation replace mapreduce.shuffle with mapreduce_shuffle
core-site.xml
<configuration> <property> <name>fs.default.name</name> <value>hdfs://localhost:9000</value> </property> </configuration>
hdfs-site.xml
<configuration> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:/home/gpuser/data/namenode</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:/home/gpuser/data/datanode</value> </property> </configuration>
If this file does not exist, create it and paste the content provided below:
mapred-site.xml
<?xml version="1.0"?> <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration>
Run Hadoop
Initialize the Hadoop File system
Namenode needs to be initialized before starting with Hadoop File System. Once intialized, Namenode creates a unique namespace id for this instance of the Hadoop File System.
- Create the Storage Directories for HDFS
Storage Directories for HDFS
$mkdir -p $HOME/data/namenode $mkdir -p $HOME/data/datanode
Initialize the filesystem
gpuser@ubuntu:~/hadoop-2.0.3-alpha$ bin/hadoop namenode -format o/p: Formatting using clusterid: CID-8b844021-d1ea-4b0a-a625-d17dcc133299 13/04/04 02:07:39 INFO util.HostsFileReader: Refreshing hosts (include/exclude) list 13/04/04 02:07:39 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000 13/04/04 02:07:39 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false 13/04/04 02:07:39 INFO blockmanagement.BlockManager: defaultReplication = 1 13/04/04 02:07:39 INFO blockmanagement.BlockManager: maxReplication = 512 13/04/04 02:07:39 INFO blockmanagement.BlockManager: minReplication = 1 13/04/04 02:07:39 INFO blockmanagement.BlockManager: maxReplicationStreams = 2 13/04/04 02:07:39 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks = false 13/04/04 02:07:39 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000 13/04/04 02:07:39 INFO blockmanagement.BlockManager: encryptDataTransfer = false 13/04/04 02:07:39 INFO namenode.FSNamesystem: fsOwner = gpuser (auth:SIMPLE) 13/04/04 02:07:39 INFO namenode.FSNamesystem: supergroup = supergroup 13/04/04 02:07:39 INFO namenode.FSNamesystem: isPermissionEnabled = true 13/04/04 02:07:39 INFO namenode.FSNamesystem: HA Enabled: false 13/04/04 02:07:39 INFO namenode.FSNamesystem: Append Enabled: true 13/04/04 02:07:40 INFO namenode.NameNode: Caching file names occuring more than 10 times 13/04/04 02:07:40 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033 13/04/04 02:07:40 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0 13/04/04 02:07:40 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000 13/04/04 02:07:41 INFO common.Storage: Storage directory /home/gpuser/data/namenode has been successfully formatted. 13/04/04 02:07:41 INFO namenode.FSImage: Saving image file /home/gpuser/data/namenode/current/fsimage.ckpt_0000000000000000000 using no compression 13/04/04 02:07:41 INFO namenode.FSImage: Image file of size 121 saved in 0 seconds. 13/04/04 02:07:41 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0 13/04/04 02:07:41 INFO util.ExitUtil: Exiting with status 0 13/04/04 02:07:41 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at ubuntu/<IP_ADDRESS> ************************************************************ gpuser@ubuntu:~/hadoop-2.0.3-alpha$
Start Hadoop File System
Start Namenode
gpuser@ubuntu:~/hadoop-2.0.3-alpha$ sbin/hadoop-daemon.sh start namenode starting namenode, logging to /home/gpuser/hadoop-2.0.3-alpha/logs/hadoop-gpuser-namenode-ubuntu.out
gpuser@ubuntu:~/hadoop-2.0.3-alpha$ sbin/hadoop-daemon.sh start datanode starting datanode, logging to /home/gpuser/hadoop-2.0.3-alpha/logs/hadoop-gpuser-datanode-ubuntu.out
Verify that the Namenode and Datanode are running using jps
gpuser@ubuntu:~/hadoop-2.0.3-alpha$ jps 18509 Jps 17107 NameNode 17170 DataNode
Start Yarn Daemons
Start Resource Manager
gpuser@ubuntu:~/hadoop-2.0.3-alpha$ sbin/yarn-daemon.sh start resourcemanager starting resourcemanager, logging to /home/gpuser/hadoop-2.0.3-alpha/logs/yarn-gpuser-resourcemanager-ubuntu.out gpuser@ubuntu:~/hadoop-2.0.3-alpha$ jps 18509 Jps 17107 NameNode 17170 DataNode 17252 ResourceManager
Start Node Manager
gpuser@ubuntu:~/hadoop-2.0.3-alpha$ sbin/yarn-daemon.sh start nodemanager starting nodemanager, logging to /home/gpuser/hadoop-2.0.3-alpha/logs/yarn-gpuser-nodemanager-ubuntu.out
gpuser@ubuntu:~/hadoop-2.0.3-alpha$jps 18509 Jps 17107 NameNode 17170 DataNode 17252 ResourceManager 17309 NodeManager
Start Job History Server
gpuser@ubuntu:~/hadoop-2.0.3-alpha$ sbin/mr-jobhistory-daemon.sh start historyserver starting historyserver, logging to /home/gpuser/hadoop-2.0.3-alpha/logs/mapred-gpuser-historyserver-ubuntu.out gpuser@ubuntu:~/hadoop-2.0.3-alpha$jps 18509 Jps 17107 NameNode 17170 DataNode 17252 ResourceManager 17309 NodeManager 17626 JobHistoryServer
Verify Hadoop installation
With Web Interfaces.
Open the browser with the following URL’s
HDFS-UI: http://localhost:50070 Resource Mnager UI: http://localhost:8088 Job History UI: http://localhost:19888
Run the WordCount example
Verify the installation by running the Wordcount Example. This is an example to count the number of times, each word appears in the given input data set.
Create input dir and create a sample file ‘animals.txt’ with the following content. Use your favoirite editor vi or emacs or gedit.
cat dog elephant zebra wolf cat dog elephant zebra wolf cat dog elephant zebra wolf cat dog elephant zebra wolf
Copy the animals.txt file from local filesystem to a ‘/input’ file in hadoop filesystem using
gpuser@ubuntu:~/hadoop-2.0.3-alpha$ bin/hadoop dfs -copyFromLocal \ input/animals.txt /input
Run the wordcount MapReduce program
gpuser@ubuntu:~/hadoop-2.0.3-alpha$ bin/hadoop jar \ share/hadoop/mapreduce/hadoop-mapreduce-examples-2.*-alpha.jar wordcount /input /output gpuser@ubuntu:~/hadoop-2.0.3-alpha$ bin/hadoop dfs -ls / DEPRECATED: Use of this script to execute hdfs command is deprecated. Instead use the hdfs command for it. Found 3 items drwxr-xr-x - gpuser supergroup 0 2013-04-04 02:44 /output -rw-r--r-- 1 gpuser supergroup 129 2013-04-04 02:43 /input drwxrwx--- - gpuser supergroup 0 2013-04-04 02:42 /tmp
Use dfs cat command to see the output
gpuser@ubuntu:~/hadoop-2.0.3-alpha$ bin/hadoop dfs -cat /output/* cat 4 dog 4 elephant 4 zebra 4 wolf 4
Multi-node installation
Running Hadoop 2.0.3-alpha On Ubuntu Linux – Multi Node Cluster
The goal of this tutorial is to run Hadoop 2.x on a multi-node cluster. For this exercise, we would be using two Ubuntu Linux boxes.
Prerequisities
Two Linux boxes with Ubuntu 12.* or One Linux box and one VM on the same box
Running Hadoop Version 2 – Single Node cluster. Complete Singlenode Hadoop installation tutorial before proceeding further.
This tutorial has been tested with the following software versions:
Ubuntu Linux 12.04 LTS
JDK 1.7
Through out the tutorial,commands to be executed on the master and slave node are mentioned as follows:
master – execute commands on the master node
slave – execute commands on the slave node
Overview
The following picture gives a brief overview of the cluster. Two Node Cluster
FIgure shows a two node cluster. Since, we have only two nodes, we will use the master node as both master and slave. For clarity, master and slave1 are shown as separate nodes. However, in this tutorial,master will also act as one of the slave.
Master node runs both the Namenode and ResourceManager.Since Master is also a slave, NodeManger and DataNode also runs on the master node.
Master
- NameNode
- Resource Manager
- NodeManager
- HstoryServer
- DataNode
- slave1
Data Node
- Node Manager
- Steps for Installation
Step 1:Preparing the machines for the Cluster
Install the operating system on both the machines. The machines should be able to ping each other. Check the IP addresses of the machines. If dhcp is used, make sure the ip-address does not change for the rest of the tutorial.
Note the IP addresses. The IP address can be obtained by using the command
$ sudo ifconfig eth0
master – <IP_ADDRESS>
slave – <IP_ADDRESS>
*Note: IP addresses in your system may be different
Edit /etc/hosts on the both master and slave. Add the following lines in /etc/hosts file
localhost <IP_ADDRESS> <IP_ADDRESS> master <IP_ADDRESS> slave
Make sure other entries are removed or commented in the/etc/hosts file. Verify that both machines ping each other “`
master – Ping slave from master
From the master machine issue the following command
gpuser@master$ping slave
slave – Ping master from slave
From the slave machine issue the following command
gpuser@slave$ping master
The ping should be successful, if not contact your IT support to fix any networking issues.
Step 2: Creating dedicated user and storage directories
Create user gpuser(home dir /home/gpuser) and group gp on both master and slave
Create the following directories on namenode and masternode under home directory /home/gpuser
master
$mkdir -p data/namenode $mkdir -p data/datanode
slave
mkdir -p data/datanode
Step 3: Install ssh and set up password less ssh between master and slave
master
Install SSH
gpuser@master:~$sudo apt-get install ssh gpuser@master:~$sudo apt-get install rsync
Generate public/private RSA key pairs using the following command.
gpuser@master:~$ ssh-keygen -t rsa -P "" Generating public/private rsa key pair. Enter file in which to save the key (/home/gpuser/.ssh/id_rsa): Created directory '/home/gpuser/.ssh'. Your identification has been saved in /home/gpuser/.ssh/id_rsa. Your public key has been saved in /home/gpuser/.ssh/id_rsa.pub. The key fingerprint is: 3c:f7:ca:f2:49:a7:05:ee:90:66:05:0f:7b:a9:63:fc gpuser@master The key's randomart image is: +--[ RSA 2048]----+ | | | | | o | | . = . | | S B | | . O o | | X o + | | +.B * | | oE | +-----------------+
Press Enter to save the key in the default location.
Enable ssh to the local machine using the following command.
gpuser@master:~$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
Test the ssh setup by connecting to the local machine with the gpuser user.
gpuser@master:~$ ssh localhost The authenticity of host 'localhost (<IP_ADDRESS>)' can't be established. ECDSA key fingerprint is ca:96:1c:5a:38:f8:9f:99:45:d4:57:82:3c:1b:64:b7. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts. The programs included with the Ubuntu system are free software; the exact distribution terms for each program are described in the
slave
repeat the above steps on slave
Authorize master to allow ssh on slave node without the password
master
gpuser@master~$ ssh-copy-id -i $HOME/.ssh/id_rsa.pub gpuser@slave
Execute the command ssh slave to verify ssh to slave works without the password
gpuser@master:~$ ssh slave Welcome to Ubuntu 12.10 (GNU/Linux 3.5.0-17-generic i686) * Documentation: https://help.ubuntu.com/ 329 packages can be updated. 98 updates are security updates. Last login: Fri Apr 5 00:32:00 2013 from ip6-localhost gpuser@slave:~$ #exit from the ssh gpuser@slave:~$ exit
Step 4: Install Hadoop on Master
Follow the steps as outlined in the tutorial ‘Running Hadoop Version 2 – Single Node cluster’ and complete the installation of Hadoop on master.
Step 5: Master node configuration
Set the following environement variables in .bashrc
Open .bashrc file in the home folder and add the following lines at the end
export HADOOP_HOME=$HOME/hadoop-2.0.3-alpha export HADOOP_MAPRED_HOME=$HOME/hadoop-2.0.3-alpha export HADOOP_COMMON_HOME=$HOME/hadoop-2.0.3-alpha export HADOOP_HDFS_HOME=$HOME/hadoop-2.0.3-alpha export YARN_HOME=$HOME/hadoop-2.0.3-alpha export HADOOP_CONF_DIR=$HOME/hadoop-2.0.3-alpha/etc/hadoop export JAVA_HOME=$HOME/java/jdk1.7.0_17 export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin
Source the variables using the following command
$ source ~/.bashrc
Search for JAVA_HOME and set export JAVA_HOME variable in libexec/hadoop-config.sh
export JAVA_HOME=$HOME/java/jdk1.7.0_17
Add following lines at start of script in etc/hadoop/yarn-env.sh :
export JAVA_HOME=$HOME/java/jdk1.7.0_17 export HADOOP_HOME=/home/hadoop-2.0.3-alpha export HADOOP_MAPRED_HOME=$HADOOP_HOME export HADOOP_COMMON_HOME=$HADOOP_HOME export HADOOP_HDFS_HOME=$HADOOP_HOME export YARN_HOME=$HADOOP_HOME export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
Hadoop configration files at located at $HADOOP_HOME/etc/hadoop.
Update the configuration files with the following entries respectively.
$HADOOP_HOME/etc/hadoop/master
Open master file and add the following line
master
$HADOOP_HOME/etc/hadoop/slaves
Open slaves file and add the following lines
master slave
*Note that the master is also listed as one the slave.
Open the file and copy the following contents respectively.
$HADOOP_HOME/etc/hadoop/core-site.xml
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>fs.default.name</name> <value>hdfs://master:9000</value> </property> </configuration>
$HADOOP_HOME/etc/hadoop/hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>dfs.replication</name> <value>2</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:/home/gpuser/data/namenode</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:/home/gpuser/data/datanode</value> </property> </configuration>
$HADOOP_HOME/etc/hadoop/mapred-site.xml
<?xml version="1.0"?> <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration>
$HADOOP_HOME/yarn-site.xml :
<?xml version="1.0"?> <configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce.shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>master:8030</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>master:8031</value> </property> <property> <name>yarn.resourcemanager.address</name> <value>master:8032</value> </property> </configuration>
Step 6: Install hadoop on Slave node
master
Use the following command to copy hadoop installation from master to the slave
gpuser@master:~$ scp -r hadoop-2.0.3-alpha gpuser@slave:/home/gpuser
slave
Verify that the following folder exists on slave.
gpuser@slave:~$ ls -al hadoop-2.0.3-alpha
Step 7: Slave node configuration
Open .bashrc file in the home folder and add the following lines at the end
export HADOOP_HOME=$HOME/hadoop-2.0.3-alpha export HADOOP_MAPRED_HOME=$HOME/hadoop-2.0.3-alpha export HADOOP_COMMON_HOME=$HOME/hadoop-2.0.3-alpha export HADOOP_HDFS_HOME=$HOME/hadoop-2.0.3-alpha export YARN_HOME=$HOME/hadoop-2.0.3-alpha export HADOOP_CONF_DIR=$HOME/hadoop-2.0.3-alpha/etc/hadoop export JAVA_HOME=$HOME/java/jdk1.7.0_17 export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin
Source the variables using the following command
$ source ~/.bashrc
Search for JAVA_HOME and set export JAVA_HOME variable in libexec/hadoop-config.sh
export JAVA_HOME=$HOME/java/jdk1.7.0_17
Search for JAVA_HOME and set export JAVA_HOME variable in etc/hadoop/yarn-env.sh
export JAVA_HOME=$HOME/java/jdk1.7.0_17
Remove the file $HADOOP_HOME/etc/hadoop/slaves
slave
rm $HADOOP_HOME/etc/hadoop/slaves
open the file and copy the following contents respectively
yarn-site.xml
<?xml version="1.0"?> <configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce.shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>master:8030</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>master:8031</value> </property> <property> <name>yarn.resourcemanager.address</name> <value>master:8032</value> </property> </configuration>
core-site.xml
<?xml version="1.0"?> <configuration> <property> <name>fs.default.name</name> <value>hdfs://master:9000</value> </property> </configuration>
hdfs-site.xml
<?xml version="1.0"?> <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:/home/gpuser/data/namenode</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:/home/gpuser/data/datanode</value> </property> </configuration>
mapred-site.xml
<?xml version="1.0"?> <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration>
Step 8: Initialize Hadoop File System
Namenode needs to be initialized before starting with Hadoop File System. Namenode creates a unique namespace id for this instance of the Hadoop File System.
gpuser@master:~/hadoop-2.0.3-alpha$ bin/hadoop namenode -format Formatting using clusterid: CID-8b844021-d1ea-4b0a-a625-d17dcc133299 13/04/04 02:07:39 INFO util.HostsFileReader: Refreshing hosts (include/exclude) list 13/04/04 02:07:39 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000 13/04/04 02:07:39 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false 13/04/04 02:07:39 INFO blockmanagement.BlockManager: defaultReplication = 1 13/04/04 02:07:39 INFO blockmanagement.BlockManager: maxReplication = 512 13/04/04 02:07:39 INFO blockmanagement.BlockManager: minReplication = 1 13/04/04 02:07:39 INFO blockmanagement.BlockManager: maxReplicationStreams = 2 13/04/04 02:07:39 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks = false 13/04/04 02:07:39 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000 13/04/04 02:07:39 INFO blockmanagement.BlockManager: encryptDataTransfer = false 13/04/04 02:07:39 INFO namenode.FSNamesystem: fsOwner = gpuser (auth:SIMPLE) 13/04/04 02:07:39 INFO namenode.FSNamesystem: supergroup = supergroup 13/04/04 02:07:39 INFO namenode.FSNamesystem: isPermissionEnabled = true 13/04/04 02:07:39 INFO namenode.FSNamesystem: HA Enabled: false 13/04/04 02:07:39 INFO namenode.FSNamesystem: Append Enabled: true 13/04/04 02:07:40 INFO namenode.NameNode: Caching file names occuring more than 10 times 13/04/04 02:07:40 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033 13/04/04 02:07:40 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0 13/04/04 02:07:40 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000 13/04/04 02:07:41 INFO common.Storage: Storage directory /home/gpuser/data/namenode has been successfully formatted. 13/04/04 02:07:41 INFO namenode.FSImage: Saving image file /home/gpuser/data/namenode/current/fsimage.ckpt_0000000000000000000 using no compression 13/04/04 02:07:41 INFO namenode.FSImage: Image file of size 121 saved in 0 seconds. 13/04/04 02:07:41 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0 13/04/04 02:07:41 INFO util.ExitUtil: Exiting with status 0 13/04/04 02:07:41 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at ubuntu/<IP_ADDRESS> ************************************************************ gpuser@master:~/hadoop-2.0.3-alpha$
Step 9: Start HDFS on Master Node
$ sbin/hadoop-daemon.sh start namenode $ sbin/hadoop-daemons.sh start datanode
Verify namenode and data node running on master and slave
Check namenode and datanode running using the following command
master
$ jps 19785 DataNode 24567 Jps 16407 NameNode
slave
Check datanode running using the following command
$ jps 13589 DataNode 15203 Jps
Step 10: Start Yarn Services
Run the following commands on the master node
$ sbin/yarn-daemon.sh start resourcemanager $ sbin/yarn-daemons.sh start nodemanager $ sbin/mr-jobhistory-daemon.sh start historyserver
Verify yarn services running in master node
master
$ jps 19785 DataNode 24567 Jps 16407 NameNode 16409 ResourceManager 16410 NodeManager 16542 HistoryServer
Verify yarn services running in slave node
slave
$ jps 13589 DataNode 13520 NodeManager 15203 Jps
Verify Hadoop installation
With Web Interfaces.
Open the browser with the following URL’s
HDFS-UI: http://localhost:50070 HDFS-UI Resource Mnager UI: http://localhost:8088 ResourceManager Job History UI: http://localhost:19888 NodeManager Run the WordCount example
Verify the installation by running the Wordcount Example. This is an example to count the number of times, each word appears in the given input data set. Create input dir and create a sample file ‘animals.txt’ with the following content. Use your favoirite editor vi or emacs or gedit.
cat dog elephant zebra wolf cat dog elephant zebra wolf cat dog elephant zebra wolf cat dog elephant zebra wolf
Copy the animals.txt file from local filesystem to a ‘/input’ file in hadoop filesystem using
gpuser@master:~/hadoop-2.0.3-alpha$ bin/hadoop dfs -copyFromLocal /home/animals.txt /input
Run the wordcount MapReduce program
gpuser@master:~/hadoop-2.0.3-alpha$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.*-alpha.jar wordcount /input /output gpuser@master:~/hadoop-2.0.3-alpha$ bin/hadoop dfs -ls / DEPRECATED: Use of this script to execute hdfs command is deprecated. Instead use the hdfs command for it. Found 3 items drwxr-xr-x - gpuser supergroup 0 2013-04-04 02:44 /output -rw-r--r-- 1 gpuser supergroup 129 2013-04-04 02:43 /input drwxrwx--- - gpuser supergroup 0 2013-04-04 02:42 /tmp
Use dfs cat command to see the output
gpuser@master:~/hadoop-2.0.3-alpha$ bin/hadoop dfs -cat /output/* cat 4 dog 4 elephant 4 zebra 4 wolf 4
HDFS
Agenda of the tutorial is to give the insights to Hadoop Distributed File System.
Introduction to HDFS
The foundation of efficient data processing in Hadoop is its data storage model.HDFS is Hadoop’s implementation of a distributed filesystem. It is designed to hold a large amount of data, and provide access to this data to many clients distributed across a network.To be able to successfully leverage HDFS, you first must understand how it is implemented and how it works.
The Design of HDFS
The HDFS design is based on the design of the Google File System (GFS).
HDFS is a filesystem designed for storing very large files with streaming data access patterns, running on clusters of commodity hardware.
Very large files
“Very large” in this context means files that are hundreds of megabytes, gigabytes, or terabytes in size.
Streaming data access
HDFS uses “write-once, read-many-times” pattern .Over the time we are performing several operation on data which is once copied to HDFS.In this case the total amout of time took to access whole data is more important than the latency in reading the first record.
Commadity hardware
Hadoop can run on commonly available hardware.
Low-latency data access
Applications that require low-latency access to data, in the tens of milliseconds range, will not work well with HDFS.
Lots of small files
number of files in a filesystem is governed by the amount of memory on the namenode. As a rule of thumb, each file, directory, and block takes about 150 bytes. So, for example, if you had one million files, each taking one block, you would need at least 300 MB of memory. While storing millions of files is feasible, billions is beyond the capability of current hardware
Multiple writers,arbitary file modifications
Files in HDFS may be written to by a single writer. Writes are always made at the end of the file.
HDFS Concepts
Blocks
A block size is the minimum amout of data that can be read or write in to disk.
File system blocks are typically a few killo bytes in size.
HDFS, too, has the concept of a block, but it is a much larger unit—64 MB by default.
Files in HDFS are broken into block-sized chunks,which are stored as independent units.
HDFS block size is large to minimize the cost of seeks.
Advantages of block abstraction:
1)A file can be larger than any single disk in network.
2)Since unit of abstraction a block rather than a file simplifies storage subsystem.
3)Blocks fit well with replication for providing fault tolerant
Namenodes and Datanodes
Namenode manages the filesystem name space.
Namenode maintains filesystem tree and metadata of all the files in directories in the tree.This information is stored in namespace image and edit log.
Data nodes are work horses of file system.They store and retrieve blocks when they are told by client.Data nodes report back to name node periodically with list of blocks that they are storing.
Secondary namenode periodically merge the name space image with edit log to prevent edit log from becoming too large.
The secondary namenode usually runs on a separate physical machine
It keeps a copy of the merged name space image, which can be used in the event of the namenode failing.
In case of total failure of the primary, data loss is almost certain. we can only copy the namenode’s metadata files that are on NFS to the secondary and run it as the new primary.
HDFS fedaration
Namenode keeps log of every file and block in the filesystem in memory.So the memory available on namenode is the limiting factor for scaling.
Scaling Namenode can be achieved by HDFS fedaration.
In federation, each namenode manages a namespace volume, which is made up of the metadata for the namespace, and a block pool containing all the blocks for the files in the namespace.
Namespace volumes are independent of each other, furthermore the failure of one namenode does not affect the availability of the namespaces managed by other namen- odes.
HDFS High Availability
- High availability for the HDFS NameNode, which eliminates the previous SPOF in HDFS.
- Support for filesystem snapshots in HDFS, which brings native backup and disaster recovery processes to Hadoop.
- Support for federated NameNodes, which allows for horizontal scaling of the filesystem namespace.
- Support for NFS access to HDFS, which allows HDFS to be mounted as a standard filesystem.
Try HDFS Commands
Hadoop provides several ways of accessing HDFS. The FileSystem (FS) shell commands provide a wealth of operations supporting access and manipulation of HDFS files. These operations include viewing HDFS directories,creating files,deleting iles,copying files, and so on.
In this document we will try various HDFS Commands. Practising these commands helps one to understand and use Hadoop better.
hadoop fs help information
[esammer@hadoop01 ~]$ hadoop fs Usage: java FsShell [-ls <path>] [-lsr <path>] [-df [<path>]] [-du <path>] [-dus <path>] [-count[-q] <path>] [-mv <src> <dst>] [-cp <src> <dst>] [-rm [-skipTrash] <path>] [-rmr [-skipTrash] <path>] [-expunge] [-put <localsrc> … <dst>] [-copyFromLocal <localsrc> … <dst>] [-moveFromLocal <localsrc> … <dst>] [-get [-ignoreCrc] [-crc] <src> <localdst>] [-getmerge <src> <localdst> [addnl]] [-cat <src>] [-text <src>] [-copyToLocal [-ignoreCrc] [-crc] <src> <localdst>] [-moveToLocal [-crc] <src> <localdst>] [-mkdir <path>] [-setrep [-R] [-w] <rep> <path/file>] [-touchz <path>] [-test -[ezd] <path>] [-stat [format] <path>] [-tail [-f] <file>] [-chmod [-R] <MODE[,MODE]… | OCTALMODE> PATH…] [-chown [-R] [OWNER][:[GROUP]] PATH…] [-chgrp [-R] GR
Try the following commands
Running Hadoop in Local Mode
If you already have configured hadoop to run in Psuedo mode or cluster mode, use the following file to point to the right hadoop cluster configuration
#Lists all the files in the local file system
hadoop fs -conf /tmp/hadoop-local.xml -ls /
#Lists all the files in the HDFS file system
hadoop fs -conf /tmp/hadoop-cluster.xml -ls /
#Lists all the files in the $HADOOP_HOME/conf/hdfs-site.xml
#uses the configuration file found in the class path
Running Hadoop in Pusedo distributed Mode
-
Hadoop must be running in Psuedo or cluster mode
hadoop fs -ls / bin/hadoop fs -mkdir stocks bin/hadoop fs -ls /user/hadoop/stocks
#copy file from local file system to hadoop file system
hadoop fs -copyFromLocal conf hdfs://localhost:9000/user/hadoop
hadoop fs -copyFromLocal conf/* hdfs://localhost:9000/user/hadoop
#copy the file to local diectory
bin/hadoop dfs -copyToLocal /usr/hadoop/conf /conf
#rm a directory
bin/hadoop fs -rmr stocks
#remove a filename
bin/hadoop fs -rm filename
#expunge empty the trash
hadoop dfs -expunge
#Get files to the local file system
hadoop dfs -get /hadooptmp/input /tmp
#cp from src to dest
hadoop dfs -cp src dest
#disk usage
hadoop dfs -dus filename
#Dir of files ; give the total size of the dir
hadoop dfs -du dir
#recirvsive version of ls
bin/hadoop dfs -lsr /
#disctp for copying files from one hdfs file system to the other
hadoop distcp hdfs://namenode1://foo hdfs://namenode2/foo
#skils if the file is already present use -overwrite if need to be overwirtten
#betweeb two versions hdfs:// will not work; use http:// or webhdfs
#move files from dir1 to dir2
bin/hadoop dfs -mv /dir1 /dir2
#copy from local to hdfs
bin/hadoop dfs -put /hadooptmp/programming_hive.pdf
#remove a file
hadoop dfs rm filename
#set replication factor to 3
bin/hadoop dfs -setrep -w 3 -R /dir2
#Returns the stat information on the path.
bin/hadoop dfs -stat /dir2
#test the existance of the file
bin/hadoop dfs -test -[ezd] URI
#create an emptyfile
bin/hadoop -touchz pathname
#output the fil in text format
bin/hadoop dfs -text /input/ncdc/sample.txt
#Displays last kilobyte of the file to stdout. -f option can be used as in Unix.
bin/hadoop dfs -tail pathname
#Adding new data nodes and tak tsacker nodes
#add hostname to dfs.hosts
#add hostname to mapred.hosts
#adding a new node
hadoop dfsadmin -refreshNodes
#Adding a new task tracker
hadoop mradmin -refreshNodes
#Run the Java programs illustrating the API
#Decommisioning a node
#add hosts dfs.hosts.exclude
#add hosts to mapred.hosts.exclude
#ado this to start decommisioning process
hadoop dfsadmin -refreshNodes
#setting log levels
hadoop daemonlog -setLevel jobtracker-host:port \org.apache.hadoop.mapred.JobTracker DEBUG
#to make persistent log file
#add the following line to log4j.properties
log4j.logger.org.apache.hadoop.mapred.JobTracker=DEBUG
#cluster balancing
hadoop balancer [-threshold ]
hadoop dfs -chown [-R] user:group filenames
#chmode
hadoop dfs -chmode [-R] user:group filenames
#hadoo archives
% hadoop fs -lsr /my/files
-rw-r--r-- drwxr-xr-x -rw-r--r--
1 tom supergroup - tom supergroup 1 tom supergroup
1 2009-04-09 19:13 /my/files/a
0 2009-04-09 19:13 /my/files/dir
1 2009-04-09 19:13 /my/files/dir/b
This job produces a very small amount of output, so it is convenient to copy it from HDFS to our development machine. The -getmerge option to the hadoop fs command is useful here, as it gets all the files in the directory specified in the source pattern and merges them into a single file on the local filesystem:
% hadoop fs -getmerge max-temp max-temp-local
% sort max-temp-local | tail
1991 607
1992 605
1993 567 1994 568 1995 567 1996 561 1997 565 1998 568 1999 568 2000 558
Java-API
Access to HDFS is through an instance of the FileSystem object. A FileSystem class is an abstract base class for a generic filesystem.
We can perform the following operatins using Java -Api
The goals of this tutorial
1.HDFS Java API Introduction
2.HDFS configuration
3.Reading Data from HDFS
4.Writing Data to HDFS
5.Browsing file system
HDFS Java API Introduction
org.apache.hadoop.fs.FileSystem is an abstract class that serves as a generic file system representation.org.apache.hadoop.fs.FileSystem is a class and not an Interface.
Different types of File system Implementations
org.apache.hadoop.fs.LocalFileSystem
- Good old native file system using local disk(s)
org.apache.hadoop.hdfs.DistributedFileSystem
- Hadoop Distributed File System (HDFS)
org.apache.hadoop.hdfs.HftpFileSystem
- Access HDFS in read-only mode over HTTP
org.apache.hadoop.fs.ftp.FTPFileSystem
- File system on FTP server
Hive - Introduction
Hadoop got a challenge in terms of moving existing data infrastructures to Hadoop ecosystem.
Mapping SQL data operations to the low-level MapReduce Java API is a tedious task. Hive does this dirty work for you, so you can focus on the query itself. Hive translates most queries to MapReduce jobs, thereby exploiting the scalability of Hadoop, while presenting a familiar SQL abstraction.
Hive is best suited for data warehouse applications, where a large data set is main-
tained and mined for insights, reports.
Hive Arichtecture
Metastore: stores system catalog
To support features like schema(s) and data partitioning Hive keeps its metadata in a Relational Database.
Meta store is packaged with Derby, a lightweight embedded SQL DB.
Schema is not shared between users as each user has their own instance of embedded Derby.
Stored in metastore_db directory which resides in the directory that hive was started from
Driver: manages life cycle of HiveQL query as it moves thru’ HIVE; also manages session handle and session statistics
Query compiler: Compiles HiveQL into a directed acyclic graph of map/reduce tasks
Execution engines: The component executes the tasks in proper dependency order; interacts with Hadoop
HiveServer: provides Thrift interface and JDBC/ODBC for integrating other applications.
Client components: CLI, web interface, jdbc/odbc inteface Extensibility interface include SerDe, User Defined Functions and User Defined Aggregate Function.
Hive – Run Modes
Starting Hive
If you added $HIVE_HOME/bin to your PATH, you can just type hive to run the command.
$ cd $HIVE_HOME $ bin/hive
Hive history file=/tmp/myname/hive_job_log_myname_201201271126_1992326118.txt hive
> CREATE TABLE x (a INT); OK Time taken: 3.543 seconds hive> SELECT * FROM x; OK Time taken: 0.231 seconds hive> SELECT * > FROM x; OK Time taken: 0.072 seconds hive> DROP TABLE x; OK Time taken: 0.834 seconds hive> exit; $
Local Mode Configuration
Go to the $HIVE_HOME/conf directory. The curious may want to peek at the large hive-default.xml.template file, which shows the different configuration properties supported by Hive and their default values. Most of these properties you can safely ignore. Changes to your configuration are done by editing the hive-site.xml file. Create one if it doesn’t already exist.
Here is an example configuration file where we set several properties for local mode execution.
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>hive.metastore.warehouse.dir</name> <value>/home/me/hive/warehouse</value> <description> Local or HDFS directory where Hive keeps table contents. </description> </property> <property> <name>hive.metastore.local</name> <value>true</value> <description> Use false if a production metastore server is used. </description> </property> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:derby:;databaseName=/home/me/hive/metastore_db;create=true</value> <description> The JDBC connection URL. </description> </property> </configuration>
Distributed and Pseudodistributed Mode Configuration
In distributed mode, several services run in the cluster. The JobTracker manages jobs and the NameNode is the HDFS master. Worker nodes run individual job tasks, man- aged by a TaskTracker service on each node, and then hold blocks for files in the distributed filesystem, managed by DataNode services.
One Hive property you might want to configure is the top-level directory for table storage, which is specified by the property hive.metastore.warehouse.dir ohe default value for this property is /user/hive/warehouse in the Apache Hadoop and MapR distributions, which will be interpreted as a distributed filesystem path when Hadoop is configured for distributed or pseudodistributed mode. For Amazon Elastic MapReduce (EMR), the default value is /mnt/hive0M_N/warehouse when using Hive v0.M.N (e.g., /mnt/hive08_1/warehouse).
Specifying a different value here allows each user to define their own warehouse direc- tory, so they don’t affect other system users. Hence, each user might use the following statement to define their own warehouse directory:
set hive.metastore.warehouse.dir=/user/myname/hive/warehouse;
Metastore Using JDBC
Hive requires only one extra component that Hadoop does not already have; the metastore component. The metastore stores metadata such as table schema and parti- tion information that you specify when you run commands such as create table x…, or alter table y…, etc. Because multiple users and systems are likely to need concurrent access to the metastore, the default embedded database is not suitable for production.
Any JDBC-compliant database can be used for the metastore. In practice, most instal- lations of Hive use MySQL. We’ll discuss how to use MySQL. It is straightforward to adapt this information to other JDBC-compliant databases. For our MySQL configuration, we need to know the host and port the service is running on. We will assume db1.mydomain.pvt and port 3306, which is the standard MySQL port.
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://db1.mydomain.pvt/hive_db?createDatabaseIfNotExist=true</value> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>database_user</value> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>database_pass</value> </property> </configuration>
Hive Basics
Primitive Data Types
There are the usual built-in data types. As always, case is ignored.
| Type | Size | Literal Syntax Examples | |
|---|---|---|---|
| TINYINT | 1-byte signed integer | 20 | |
| SMALLINT | 2-byte signed integer | 20 | |
| INT | 4-byte signed integer | 20 | |
| BIGINT | 8-byte signed integer | 20 | |
| BOOLEAN | TRUE | FALSE | TRUE | |
| FLOAT | 4-byte single precision | 3.14159 | |
| DOUBLE | 8-byte double precision | 3.14159 | |
| STRING | Sequence of characters. The character set can be specified. | ‘Now is the time’, “for all good men” | |
| TIMESTAMP | Integer, float or string | (Hive v0.8)<PHONE_NUMBER> (Unix epoch seconds),<PHONE_NUMBER>.123456789 (Unix epoch seconds plus nanoseconds), and '2012-02-03 12:34:56.123456789' (JDBC-compliant java.sql.Timestamp format). | |
| BINARY | Array of bytes | (Hive v0.8) see discussion below. |
Note that these types reflect the underlying types provided by Java.
TIMESTAMP and BINARY are new to Hive v0.8.0. For Hive v0.7.X, use an integer value for the epoch seconds or use a STRING.
If you use the latter, it is best to use a sortable format like the example shown.
The BINARY type is a way of saying “ignore the rest of this record”; it is treated as a byte array without interpretation and with no built-in support for conversion to other types. Use it when you only care about the first few fields in a record, e.g.,
CREATE TABLE short (s STRING, i INT, b BINARY);
When a query mentions a particular type, Hive will implicitly cast any integer type to a larger integer type, cast FLOAT to DOUBLE, and cast any integer type to DOUBLE, as needed.
You can also explicitly interpret a STRING as a number type using, for example, '3.14159' to DOUBLE.
Complex Data Types
Hive supports columns that are structs, maps, and arrays. Note that the “literal syntax examples” are actually making calls to built-in functions.
| TYPE | DESCRIPTION | LITERAL SYNTAX EXAMPLES |
|---|---|---|
| STRUCT | Analogous to a C struct or an “object”. Fields can be accessed using the “dot” notation. For example, if a column name if of type STRUCT {first STRING; last STRING}, then the first name field can be referenced using name.first. | struct('John', 'Doe') |
| MAP | A collection of key-value tuples, where the fields are accessed using array notation, e.g., [‘key’]. For example, if a column name is of type MAP with key->value pairs 'first'->'John' and 'last'->'Doe', then the last name can be referenced using name['last']. | map('first', 'John', 'last', 'Doe') |
| ARRAY | Lists of the same type that are indexable using zero-based integers. For example, if a column name is of type ARRAY of strings with the value ['John', 'Doe'], then the second element can be referenced using name[1]. | array('John', 'Doe') |
File Formats
Hive supports all the Hadoop file formats, plus Thrift encoding, as well as supporting pluggable SerDe (serializer/deserializer) classes to support custom formats. Hive defaults to the following record and field delimiters, all of which are non-printable control characters and all of which can be customized.
| DELIMITER | NAME | USE |
|---|---|---|
| \n or \r | Line feed or carraige return | Records, i.e., one per line. |
| ^A | Control-A (\001) | Separates fields, e.g., the way commas are used in CSV files. |
| ^B | Control-B (\002) | Separates elements in the “collections” (STRUCT, MAP, and ARRAY) |
| ^C | Control-C (\003) | Separates the key and value in MAP elements. |
All of these defaults can be customized when creating tables. See examples below.
There are several file formats supported by Hive. TEXTFILE is the easiest to use, but the least space efficient. Hadoop’s SEQUENCEFILE format is more space efficient. A related format is MAPFILE which adds an index to a SEQUENCEFILE for faster retrieval of particular records.
HiveQL-Data Definition
HiveQL is the Hive query language. Like all SQL dialects in widespread use, it doesn’t fully conform to any particular revision of the ANSI SQL standard. Hive doesn’t support transactions. Hive adds ex-tensions to provide better performance in the context of Hadoop and to integrate with custom extensions and even external programs.
Prerequisites
Hadoop must be installed as per installation instructions Hadoop version 2.0.3 installation Hive must be installed as per installation instructions Hive version 0.10.0 installation Load the dataset into HDFS using this linkTables Used In this tutorial
- Stocks
- Employees
Create database
They are very useful for larger clusters with multiple teams and users, as a way of avoiding table name collisions. It’s also common to use databases to organize production tables into logical groups. If you don’t specify a database, the default database is used.
hive> CREATE DATABASE IF NOT EXISTS financials;
You can also use the keyword SCHEMA instead of DATABASE in all the database-related commands. At any time, you can see the databases that already exist as follows:
hive> SHOW DATABASES; default financials hive> CREATE DATABASE human_resources; hive> SHOW DATABASES; default financials human_resources
you can restrict the ones listed using a regular expression.
hive> SHOW DATABASES LIKE 'h.*'; human_resources
DESCRIBE DATABASE shows the directory location for the database.
hive> DESCRIBE DATABASE financials; hdfs://master-server/user/hive/warehouse/financials.db
Hive will create a directory for each database. Tables in that database will be stored in subdirectories of the database directory. The exception is tables in the default database, which doesn’t have its own directory. The database directory is created under a top-level directory specified by the property hive.metastore.warehouse.dir. Assuming you are using the default value for this property, /user/hive/warehouse, when the financials database is created, Hive will create the directory /user/hive/warehouse/financials.db. Note the .db extension.
You can override this default location for the new directory as shown in this example:
hive> CREATE DATABASE financials > LOCATION '/my/preferred/directory';
USE database
The USE command sets a database as your working database, analogous to changing working directories in a filesystem
hive> USE financials;
We can set hive.cli.print.current.db property to print the current database as part of the prompt.
hive> set hive.cli.print.current.db=true; hive (financials)> USE default; hive (default)> set hive.cli.print.current.db=false;
DROP database
you can drop a database:
hive> DROP DATABASE IF EXISTS financials;
The IF EXISTS is optional and suppresses warnings if financials doesn’t exist. By default, Hive won’t permit you to drop a database if it contains tables. You can either drop the tables first or append the CASCADE keyword to the command, which will cause the Hive to drop the tables in the database first:
hive> DROP DATABASE IF EXISTS financials CASCADE;
Alter Database
You can set key-value pairs in the DBPROPERTIES associated with a database using the ALTER DATABASE command. No other metadata about the database can be changed,including its name and directory location:
hive> ALTER DATABASE financials SET DBPROPERTIES ('edited-by' = 'active steps');
There is no way to delete or “unset” a DBPROPERTY.
Create Tables
The CREATE TABLE statement follows SQL conventions, but Hive’s version offers sig- nificant extensions to support a wide range of flexibility where the data files for tables are stored, the formats used, etc.
Managed Tables
The tables we have created so far are called managed tables or sometimes called internal tables, because Hive controls the lifecycle of their data. As we’ve seen,Hive stores the data for these tables in subdirectory under the directory defined by hive.metastore.warehouse.dir (e.g., /user/hive/warehouse), by default.
When we drop a managed table, Hive deletes the data in the table.
Managed tables are less convenient for sharing with other tools
External Tables
CREATE EXTERNAL TABLE IF NOT EXISTS stocks ( exchange STRING, symbol STRING, ymd STRING, price_open FLOAT, price_high FLOAT, price_low FLOAT, price_close FLOAT, volume INT, price_adj_close FLOAT ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION '/data/stocks/';
The EXTERNAL keyword tells Hive this table is external and the LOCATION … clause is required to tell Hive where it’s located. Because it’s external, Hive does not assume it owns the data. Therefore, dropping the table does not delete the data, although the metadata for the table will be deleted.
Partitioned, Managed Tables
Partitioned tables help to organize data in a logical fashion, such as hierarchically.
Example:Our HR people often run queries with WHERE clauses that restrict the results to a particular country or to a particular first-level subdivision (e.g., state in the United States or province in Canada).
we have to use address.state to project the value inside the address. So, let’s partition the data first by country and then by state:
CREATE TABLE IF NOT EXISTS mydb.employees ( name STRING, salary FLOAT, subordinates ARRAY<STRING>, deductions MAP<STRING, FLOAT>, address STRUCT<street:STRING, city:STRING, state:STRING, zip:INT> ) PARTITIONED BY (country STRING, state STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' COLLECTION ITEMS TERMINATED BY '\002' MAP KEYS TERMINATED BY '\003' LINES TERMINATED BY '\n' STORED AS TEXTFILE;
Partitioning tables changes how Hive structures the data storage. If we create this table in the mydb database, there will still be an employees directory for the table:
LOAD DATA LOCAL INPATH '/path/to/employees.txt' INTO TABLE employees PARTITION (country = 'US', state = 'IL'); hdfs://master_server/user/hive/warehouse/mydb.db/employees
However, Hive will now create subdirectories reflecting the partitioning structure. For example:
... .../employees/country=CA/state=AB .../employees/country=CA/state=BC ... .../employees/country=US/state=AL .../employees/country=US/state=AK
Those are the actual directory names. The state directories will contain zero or more files for the employees in those states.
Once created, the partition keys (country and state, in this case) behave like regular columns.
hive (mydb)> SHOW PARTITIONS employees; OK country=US/state=IL Time taken: 0.145 seconds hive (mydb)> SELECT * FROM employees WHERE country ='US' AND state ='IL'; OK John Doe 100000.0 ["Mary Smith","Todd Jones"] {"Federal Taxes":0.2,"State Taxes":0.05,"Insurance":0.1} {"street":"1 Michigan Ave.","city":"Chicago","state":"IL","zip":60600} US IL Mary Smith 80000.0 ["Bill King"] {"Federal Taxes":0.2,"State Taxes":0.05,"Insurance":0.1} {"street":"100 Ontario St.","city":"Chicago","state":"IL","zip":60601} US IL Todd Jones 70000.0 [] {"Federal Taxes":0.15,"State Taxes":0.03,"Insurance":0.1} {"street":"200 Chicago Ave.","city":"Oak Park","state":"IL","zip":60700} US IL Bill King 60000.0 [] {"Federal Taxes":0.15,"State Taxes":0.03,"Insurance":0.1} {"street":"300 Obscure Dr.","city":"Obscuria","state":"IL","zip":60100} US IL Boss Man 200000.0 ["John Doe","Fred Finance"] {"Federal Taxes":0.3,"State Taxes":0.07,"Insurance":0.05} {"street":"1 Pretentious Drive.","city":"Chicago","state":"IL","zip":60500} US IL Fred Finance 150000.0 ["Stacy Accountant"] {"Federal Taxes":0.3,"State Taxes":0.07,"Insurance":0.05} {"street":"2 Pretentious Drive.","city":"Chicago","state":"IL","zip":60500} US IL Stacy Accountant 60000.0 [] {"Federal Taxes":0.15,"State Taxes":0.03,"Insurance":0.1} {"street":"300 Main St.","city":"Naperville","state":"IL","zip":60563} US IL Time taken: 0.254 seconds
Partition limits the results to employees in Illinois, it is only necessary to scan the contents of one directory. Partitioning can dramatically improve query performance, but only if the partitioning scheme reflects common range filtering (e.g., by locations, timestamp ranges). When we add predicates to WHERE clauses that filter on partition values, these predicates are called partition filters.
Strict mode
A query across all partitions could trigger an enormous MapReduce job if the table data and number of partitions are large. A highly suggested safety measure is putting Hive into strict mode, which prohibits queries of partitioned tables without a WHERE clause that filters on partitions. You can set the mode to nonstrict, as in the following session:
hive> set hive.mapred.mode=strict; hive> SELECT e.name, e.salary FROM employees e LIMIT 100; FAILED: Error in semantic analysis: No partition predicate found for Alias "e" Table "employees" hive> set hive.mapred.mode=nonstrict; hive> SELECT e.name, e.salary FROM employees e LIMIT 100;
Hive Queries – Part 1
Prerequisites
- Hadoop must be installed as per installation instructions Hadoop version 2.0.3 installation
- Hive must be installed as per installation instructions Hive version 0.10.0 installation
- Load the dataset into HDFS using this link
Tables Used In this tutorial
- Employees
To create the above tables run the following ddl queries.
Create employees
CREATE EXTERNAL TABLE employees ( name STRING, salary FLOAT, subordinates ARRAY<STRING>, deductions MAP<STRING, FLOAT>, address STRUCT<street:STRING, city:STRING, state:STRING, zip:INT> ) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' COLLECTION ITEMS TERMINATED BY '\002' MAP KEYS TERMINATED BY '\003' LINES TERMINATED BY '\n' STORED AS TEXTFILE LOCATION '/data/employees';
SELECT … FROM Clauses
SELECT is the projection operator in SQL. The FROM clause identifies from which table, view, or nested query we select records
hive> SELECT name, salary FROM employees; John Doe 100000.0 Mary Smith 80000.0 Todd Jones 70000.0 Bill King 60000.0
When you select columns that are one of the collection types, Hive uses JSON (Java- Script Object Notation) syntax for the output. First, let’s select the subordinates, an ARRAY, where a comma-separated list surrounded with […] is used. Note that STRING elements of the collection are quoted, while the primitive STRING name column is not
hive> SELECT name, subordinates FROM employees; John Doe ["Mary Smith","Todd Jones"] Mary Smith ["Bill King"] Todd Jones [] Bill King []
The deductions is a MAP, where the JSON representation for maps is used, namely a comma-separated list of key:value pairs, surrounded with {…}:
hive> SELECT name, deductions FROM employees; John Doe {"Federal Taxes":0.2,"State Taxes":0.05,"Insurance":0.1} Mary Smith {"Federal Taxes":0.2,"State Taxes":0.05,"Insurance":0.1} Todd Jones {"Federal Taxes":0.15,"State Taxes":0.03,"Insurance":0.1} Bill King {"Federal Taxes":0.15,"State Taxes":0.03,"Insurance":0.1} Finally, the address is a STRUCT, which is also written using the JSON map format:
hive> SELECT name, address FROM employees; John Doe {"street":"1 Michigan Ave.","city":"Chicago","state":"IL","zip":60600} Mary Smith {"street":"100 Ontario St.","city":"Chicago","state":"IL","zip":60601} Todd Jones {"street":"200 Chicago Ave.","city":"Oak Park","state":"IL","zip":60700} Bill King {"street":"300 Obscure Dr.","city":"Obscuria","state":"IL","zip":60100}
Next, let’s see how to reference elements of collections. First, ARRAY indexing is 0-based, as in Java. Here is a query that selects the first element of the subordinates array:
hive> SELECT name, subordinates[0] FROM employees; John Doe Mary Smith Mary Smith Bill King Todd Jones NULL Bill King NULL
Note that referencing a nonexistent element returns NULL. Also, the extracted STRING values are no longer quoted! To reference a MAP element, you also use ARRAY[…] syntax, but with key values instead of integer indices:
hive> SELECT name, deductions["State Taxes"] FROM employees; John Doe 0.05 Mary Smith 0.05 Todd Jones 0.03 Bill King 0.03
Finally, to reference an element in a STRUCT, you use dot notation, similar to the table_alias.column mentioned above:
hive> SELECT name, address.city FROM employees; John Doe Chicago Mary Smith Chicago Todd Jones Oak Park Bill King Obscuri LIMIT Clause
The results of a typical query can return a large number of rows. The LIMIT clause puts an upper limit on the number of rows returned:
hive> SELECT upper(name), salary, deductions["Federal Taxes"], > round(salary * (1 - deductions["Federal Taxes"])) FROM employees > LIMIT 2; JOHN DOE 100000.0 0.2 80000 MARY SMITH 80000.0 0.2 64000
Column Aliases
You can think of the previous example query as returning a new relation with new columns, some of which are anonymous results of manipulating columns in employees. It’s sometimes useful to give those anonymous columns a name, called a column alias. Here is the previous query with column aliases for the third and fourth columns returned by the query, fed_taxes and salary_minus_fed_taxes, respectively:
hive> SELECT upper(name), salary, deductions["Federal Taxes"] as fed_taxes, > round(salary * (1 - deductions["Federal Taxes"])) as salary_minus_fed_taxes > FROM employees LIMIT 2; JOHN DOE 100000.0 0.2 80000 MARY SMITH 80000.0 0.2 64000
Nested SELECT Statements
The column alias feature is especially useful in nested select statements. Let’s use the previous example as a nested query:
hive> FROM ( > SELECT upper(name), salary, deductions["Federal Taxes"] as fed_taxes, > round(salary * (1 - deductions["Federal Taxes"])) as salary_minus_fed_taxes > FROM employees > ) e > SELECT e.name, e.salary_minus_fed_taxes > WHERE e.salary_minus_fed_taxes > 70000; JOHN DOE 100000.0 0.2 80000
The previous result set is aliased as e, from which we perform a second query to select the name and the salary_minus_fed_taxes, where the latter is greater than 70,000.
CASE … WHEN … THEN Statements
The CASE … WHEN … THEN clauses are like if statements for individual columns in query results. For example:
hive> SELECT name, salary, > CASE > WHEN salary < 50000.0 THEN 'low' > WHEN salary >= 50000.0 AND salary < 70000.0 THEN 'middle' > WHEN salary >= 70000.0 AND salary < 100000.0 THEN 'high' > ELSE 'very high' > END AS bracket FROM employees; John Doe 100000.0 very high Mary Smith 80000.0 high Todd Jones 70000.0 high Bill King 60000.0 middle Boss Man 200000.0 very high Fred Finance 150000.0 very high Stacy Accountant 60000.0 middle ...
WHERE Clauses
WHERE clauses are filters; they select which records to return.WHERE clauses use predicate expressions, applying predicate operators
The predicates can reference the same variety of computations over column values that can be used in SELECT clauses. Here we adapt our previously used query involving Federal Taxes, filtering for those rows where the salary minus the federal taxes is greater than 70,000:
hive> SELECT name, salary, deductions["Federal Taxes"], > salary * (1 - deductions["Federal Taxes"]) > FROM employees > WHERE round(salary * (1 - deductions["Federal Taxes"])) > 70000; John Doe 100000.0 0.2 80000.0
This query is a bit ugly, because the complex expression on the second line is duplicated in the WHERE clause. The following variation eliminates the duplication, using a column alias, but unfortunately it’s not valid:
hive> SELECT name, salary, deductions["Federal Taxes"], > salary * (1 - deductions["Federal Taxes"]) as salary_minus_fed_taxes > FROM employees > WHERE round(salary_minus_fed_taxes) > 70000; FAILED: Error in semantic analysis: Line 4:13 Invalid table alias or column reference 'salary_minus_fed_taxes': (possible column names are: name, salary, subordinates, deductions, address)
As the error message says, we can’t reference column aliases in the WHERE clause. How- ever, we can use a nested SELECT statement:
hive> SELECT e.* FROM > (SELECT name, salary, deductions["Federal Taxes"] as ded, > salary * (1 - deductions["Federal Taxes"]) as salary_minus_fed_taxes > FROM employees) e > WHERE round(e.salary_minus_fed_taxes) > 70000; John Doe 100000.0 0.2 80000.0 Boss Man 200000.0 0.3 140000.0 Fred Finance 150000.0 0.3 105000.0
LIKE and RLIKE
You have probably seen LIKE before, a standard SQL operator. For example, the following three queries select the employee names and addresses where the street ends with Ave., the city begins with O, and the street contains Chicago:
hive> SELECT name, address.street FROM employees WHERE address.street LIKE '%Ave.'; John Doe 1 Michigan Ave. Todd Jones 200 Chicago Ave.
hive> SELECT name, address.city FROM employees WHERE address.city LIKE 'O%'; Todd Jones Oak Park Bill King Obscuria
hive> SELECT name, address.street FROM employees WHERE address.street LIKE '%Chi%'; Todd Jones 200 Chicago Ave.
A Hive extension is the RLIKE clause, which lets us use Java regular expressions, a more powerful minilanguage for specifying matches.
hive> SELECT name, address.street > FROM employees WHERE address.street RLIKE '.*(Chicago|Ontario).*'; Mary Smith 100 Ontario St. Todd Jones 200 Chicago Ave.
The string after the RLIKE keyword has the following interpretation. A period . matches any character and a star * means repeat the “thing to the left” (period, in the two cases shown) zero to many times. The expression x|y means match either x or y. Hence, there might be no characters before “Chicago” or “Ontario” and there might be no characters after them.
Hive Queries – Part 2
Hive supports the classic SQL JOIN statement, but only equi-joins are supported.
Prerequisites
- Hadoop must be installed as per installation instructions Hadoop version 2.0.3 installation
- Hive must be installed as per installation instructions Hive version 0.10.1 installation
- Load the dataset into HDFS using this link
Data set overview
Historical NYSE stock data from 1970 – 2010, including daily open, close, low, high and trading volume figures. Data is organized alphabetically by ticker symbol.
Tables Used In this tutorial
- Dividends
- Stocks
- Employees
To create the above tables run the following ddl queries.
 HAVING is a way of further filtering the output.
SELECT year(ymd), avg(price_close) FROM stocks WHERE symbol = 'AAPL' AND exchange = 'NASDAQ' GROUP BY year(ymd) HAVING avg(price_close) > 50.0 AND avg(price_close) < 100.0;
Dividends
Create dividends
CREATE TABLE IF NOT EXISTS dividends ( ymd STRING, dividend FLOAT ) PARTITIONED BY (exchange STRING, symbol STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION '/data/dividends';
Load Dividend Data
LOAD DATA LOCAL INPATH '/path/to/data/dividends/input/plain-text/NASDAQ/AAPL/dividends.csv' INTO TABLE DIVIDENDS PARTITION (exchange = 'NASDAQ', symbol = 'AAPL'); LOAD DATA LOCAL INPATH '/path/to/data/dividends/input/plain-text/NASDAQ/INTC/dividends.csv' INTO TABLE DIVIDENDS PARTITION (exchange = 'NASDAQ', symbol = 'INTC'); LOAD DATA LOCAL INPATH '/path/to/data/dividends/input/plain-text/NYSE/GE/dividends.csv' INTO TABLE DIVIDENDS PARTITION (exchange = 'NYSE', symbol = 'GE'); LOAD DATA LOCAL INPATH '/path/to/data/dividends/input/plain-text/NYSE/IBM/dividends.csv' INTO TABLE DIVIDENDS PARTITION (exchange = 'NYSE', symbol = 'IBM');
Queries on Dividends
display 10 records of dividends table
hive> SELECT * FROM dividends LIMIT 10; display 10 records of stocks table hive> SELECT * FROM stocks LIMIT 10; Inner JOIN
In an inner JOIN, records are discarded unless join criteria finds matching records in every table being joined. For example, the following query compares Apple (symbol AAPL) and IBM (symbol IBM). The stocks table is joined against itself, a self-join, where the dates, ymd (year-month-day) values must be equal in both tables. We say that the ymd columns are the join keys in this query:
hive> SELECT a.ymd, a.price_close, b.price_close > FROM stocks a JOIN stocks b ON a.ymd = b.ymd > WHERE a.symbol = 'AAPL' AND b.symbol = 'IBM'; 2010-01-04 214.01 132.45 2010-01-05 214.38 130.85 2010-01-06 210.97 130.0 2010-01-07 210.58 129.55 2010-01-08 211.98 130.85 2010-01-11 210.11 129.48 ...
The ON clause specifies the conditions for joining records between the two tables. The WHERE clause limits the lefthand table to AAPL records and the righthand table to IBM records. You can also see that using table aliases for the two occurrences of stocks is essential in this query.
As you may know, IBM is an older company than Apple. It has been a publicly traded stock for much longer than Apple. However, since this is an inner JOIN, no IBM records will be returned older than September 7, 1984, which was the first day that Apple was publicly traded!
Standard SQL allows a non-equi-join on the join keys, such as the following example that shows Apple versus IBM, but with all older records for Apple paired up with each day of IBM data. It would be a lot of data
SELECT a.ymd, a.price_close, b.price_close FROM stocks a JOIN stocks b ON a.ymd <= b.ymd WHERE a.symbol = 'AAPL' AND b.symbol = 'IBM';
This is not valid in Hive, primarily because it is difficult to implement these kinds of joins in MapReduce. Also, Hive does not currently support using OR between predicates in ON clauses.
Here is an inner JOIN between stocks and dividends for Apple, where we use the ymd and symbol columns as join keys:
hive> SELECT s.ymd, s.symbol, s.price_close, d.dividend > FROM stocks s JOIN dividends d ON s.ymd = d.ymd AND s.symbol = d.symbol > WHERE s.symbol = 'AAPL'; 1987-05-11 AAPL 77.0 0.015 1987-08-10 AAPL 48.25 0.015 1987-11-17 AAPL 35.0 0.02 ... 1995-02-13 AAPL 43.75 0.03 1995-05-26 AAPL 42.69 0.03 1995-08-16 AAPL 44.5 0.03 1995-11-21 AAPL 38.63 0.03
You can join more than two tables together. Let’s compare Apple, IBM, and GE side by side:
hive> SELECT a.ymd, a.price_close, b.price_close , c.price_close > FROM stocks a JOIN stocks b ON a.ymd = b.ymd > JOIN stocks c ON a.ymd = c.ymd > WHERE a.symbol = 'AAPL' AND b.symbol = 'IBM' AND c.symbol = 'GE'; 2010-01-04 214.01 132.45 15.45 2010-01-05 214.38 130.85 15.53 2010-01-06 210.97 130.0 15.45 2010-01-07 210.58 129.55 16.25 2010-01-08 211.98 130.85 16.6 2010-01-11 210.11 129.48 16.76 ...
Most of the time, Hive will use a separate MapReduce job for each pair of things to join. In this example, it would use one job for tables a and b, then a second job to join the output of the first join with c.
Join Optimizations
When joining three or more tables, if every ON clause uses the same join key, a single MapReduce job will be used.
Hive also assumes that the last table in the query is the largest. It attempts to buffer the other tables and then stream the last table through, while performing joins on individual records. Therefore, you should structure your join queries so the largest table is last. Recall our previous join between stocks and dividends. We actually made the mistake of using the smaller dividends table last:
SELECT s.ymd, s.symbol, s.price_close, d.dividend FROM stocks s JOIN dividends d ON s.ymd = d.ymd AND s.symbol = d.symbol WHERE s.symbol = 'AAPL'; We should switch the positions of stocks and dividends: SELECT s.ymd, s.symbol, s.price_close, d.dividend FROM dividends d JOIN stocks s ON s.ymd = d.ymd AND s.symbol = d.symbol WHERE s.symbol = 'AAPL';
Fortunately, you don’t have to put the largest table last in the query. Hive also provides a “hint” mechanism to tell the query optimizer which table should be streamed:
SELECT /*+ STREAMTABLE(s) */ s.ymd, s.symbol, s.price_close, d.dividend FROM stocks s JOIN dividends d ON s.ymd = d.ymd AND s.symbol = d.symbol WHERE s.symbol = 'AAPL';
Now Hive will attempt to stream the stocks table, even though it’s not the last table in the query.
LEFT OUTER JOIN
The left-outer join is indicated by adding the LEFT OUTER keywords:
hive> SELECT s.ymd, s.symbol, s.price_close, d.dividend > FROM stocks s LEFT OUTER JOIN dividends d ON s.ymd = d.ymd AND s.symbol = d.symbol > WHERE s.symbol = 'AAPL'; ... 1987-05-01 AAPL 80.0 NULL 1987-05-04 AAPL 79.75 NULL 1987-05-05 AAPL 80.25 NULL 1987-05-06 AAPL 80.0 NULL 1987-05-07 AAPL 80.25 NULL 1987-05-08 AAPL 79.0 NULL 1987-05-11 AAPL 77.0 0.015 1987-05-12 AAPL 75.5 NULL 1987-05-13 AAPL 78.5 NULL 1987-05-14 AAPL 79.25 NULL 1987-05-15 AAPL 78.25 NULL 1987-05-18 AAPL 75.75 NULL 1987-05-19 AAPL 73.25 NULL 1987-05-20 AAPL 74.5 NULL ...
In this join, all the records from the lefthand table that match the WHERE clause are returned. If the righthand table doesn’t have a record that matches the ON criteria, NULL is used for each column selected from the righthand table. Hence, in this result set, we see that the every Apple stock record is returned and the d.dividend value is usually NULL, except on days when a dividend was paid (May 11th, 1987, in this output).
OUTER JOIN Gotcha
RIGHT OUTER JOIN
Right-outer joins return all records in the righthand table that match the WHERE clause. NULL is used for fields of missing records in the lefthand table. Here we switch the places of stocks and dividends and perform a righthand join, but leave the SELECT statement unchanged:
hive> SELECT s.ymd, s.symbol, s.price_close, d.dividend > FROM dividends d RIGHT OUTER JOIN stocks s ON d.ymd = s.ymd AND d.symbol = s.symbol > WHERE s.symbol = 'AAPL'; ... 1987-05-07 AAPL 80.25 NULL 1987-05-08 AAPL 79.0 NULL 1987-05-11 AAPL 77.0 0.015 1987-05-12 AAPL 75.5 NULL 1987-05-13 AAPL 78.5 NULL ...
LEFT SEMI-JOIN
A left semi-join returns records from the lefthand table if records are found in the right- hand table that satisfy the ON predicates. It’s a special, optimized case of the more general inner join. Most SQL dialects support an IN … EXISTS construct to do the same thing. For instance, the following query in Example 6-2 attempts to return stock records only on the days of dividend payments, but it doesn’t work in Hive. Example 6-2. Query that will not work in Hive SELECT s.ymd, s.symbol, s.price_close FROM stocks s WHERE s.ymd, s.symbol IN (SELECT d.ymd, d.symbol FROM dividends d); Instead, you use the following LEFT SEMI JOIN syntax:
hive> SELECT s.ymd, s.symbol, s.price_close > FROM stocks s LEFT SEMI JOIN dividends d ON s.ymd = d.ymd AND s.symbol = d.symbol; ... 1962-11-05 IBM 361.5 1962-08-07 IBM 373.25 1962-05-08 IBM 459.5 1962-02-06 IBM 551.5
Note that the SELECT and WHERE clauses can’t reference columns from the righthand table.
The reason semi-joins are more efficient than the more general inner join is as follows. For a given record in the lefthand table, Hive can stop looking for matching records in the righthand table as soon as any match is found. At that point, the selected columns from the lefthand table record can be projected.
Cartesian Product JOINs
A Cartesian product is a join where all the tuples in the left side of the join are paired with all the tuples of the right table. If the left table has 5 rows and the right table has 6 rows, 30 rows of output will be produced:
SELECT * FROM stocks JOIN dividends;
Cartesian products create a lot of data. Unlike other join types, Cartesian products are not executed in parallel, and they are not optimized in any way using MapReduce.
Map-side Joins
If all but one table is small, the largest table can be streamed through the mappers while the small tables are cached in memory. Hive can do all the joining map-side, since it can look up every possible match against the small tables in memory, thereby eliminating the reduce step required in the more common join scenarios.
The joins between stocks and dividends can exploit this optimization, as the dividends data set is small enough to be cached.
set hive.auto.convert.join, to true before Hive will attempt the optimization. It’s false by default:
hive> set hive.auto.convert.join=true; hive> SELECT s.ymd, s.symbol, s.price_close, d.dividend > FROM stocks s JOIN dividends d ON s.ymd = d.ymd AND s.symbol = d.symbol > WHERE s.symbol = 'AAPL';
Note that you can also configure the threshold size for table files considered small enough to use this optimization.
Hive does not support the optimization for right- and full-outer joins.
Map-side joins can be done on big tables when data for every table is bucketed.
Map-side join can be enabled by setting the property
hive.optimize.bucketmapjoin: set hive.optimize.bucketmapjoin=true;
If the bucketed tables actually have the same number of buckets and the data is sorted by the join/bucket keys, then Hive can perform an even faster sort-merge join. Once again, properties must be set to enable the optimization:
set hive.input.format=org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat set hive.optimize.bucketmapjoin=true set hive.optimize.bucketmapjoin.sortedmerge=true
ORDER BY and SORT BY
ORDER BY performs a total ordering of the query result set. This means that all the data is passed through a single reducer, which may take an unacceptably long time to execute for larger data sets.
Hive adds an alternative, SORT BY, that orders the data only within each reducer, thereby performing a local ordering, where each reducer’s output will be sorted. Better perfor- mance is traded for total ordering. In both cases, the syntax differs only by the use of the ORDER or SORT keyword. You can specify any columns you wish and specify whether or not the columns are ascending using the ASC keyword (the default) or descending using the DESC keyword. Here is an example using ORDER BY:
SELECT s.ymd, s.symbol, s.price_close FROM stocks s ORDER BY s.ymd ASC, s.symbol DESC;
Here is the same example using SORT BY instead:
SELECT s.ymd, s.symbol, s.price_close FROM stocks s SORT BY s.ymd ASC, s.symbol DESC;
The two queries look almost identical, but if more than one reducer is invoked, the output will be sorted differently. While each reducer’s output files will be sorted, the data will probably overlap with the output of other reducers. Because ORDER BY can result in excessively long run times, Hive will require a LIMIT clause with ORDER BY if the property hive.mapred.mode is set to strict. By default, it is set to nonstrict.
DISTRIBUTE BY with SORT BY
DISTRIBUTE BY controls how map output is divided among reducers. All data that flows through a MapReduce job is organized into key-value pairs.MapReduce computes a hash on the keys output by mappers and tries to evenly distribute the key-value pairs among the available reducers using the hash values. this means that when we use SORT BY, the contents of one reducer’s output will overlap significantly with the output of the other reducers, as far as sorted order is concerned, even though the data is sorted within each reducer’s output.
Say we want the data for each stock symbol to be captured together. We can use DISTRIBUTE BY to ensure that the records for each stock symbol go to the same reducer, then use SORT BY to order the data the way we want. The following query demonstrates this technique:
hive> SELECT s.ymd, s.symbol, s.price_close > FROM stocks s > DISTRIBUTE BY s.symbol > SORT BY s.symbol ASC, s.ymd ASC; 1984-09-07 AAPL 26.5 1984-09-10 AAPL 26.37 1984-09-11 AAPL 26.87 1984-09-12 AAPL 26.12 1984-09-13 AAPL 27.5 1984-09-14 AAPL 27.87 1984-09-17 AAPL 28.62 1984-09-18 AAPL 27.62 1984-09-19 AAPL 27.0 1984-09-20 AAPL 27.12 ...
DISTRIBUTE BY works similar to GROUP BY in the sense that it controls how reducers receive rows for processing, while SORT BY controls the sorting of data inside the reducer.


Currently working at an automotive parts manufacturer as a Quality Inspector. I work closely with engineers and machine technicians to ensure that products are free of defects and meet customer standards. I am constantly going above and beyond at DENSO by looking for ways to improve production rate and worker ergonmoics. My current project consists of working with my supervisor to implement a new training program to reduce machine downtime and increase production rate.
Worked as a lifeguard for a various pools across the Mississauga and Brampton area. Job role consisted of organizing swimming events while maintaing safety and enforcing safety protocol. As a lifeguard I was trained in CPR and Standard First Aid and carried out emergency procedures in a high stress environment. In addition, I was also responsible for maintaining safe pool water levels; I conducted water tests frequently and adjusted pool chemicals as needed.
As an advanced swim instructor, I was responsible for teaching students of all ages and skills about water safety and how to swim. In addition, I also taught various lifesaving and leadership courses to aspiring lifeguards. Working as an instructor helped me develop key customer service skills and strong teamwork. These skills are valuable and applicable to any workplace.


Directive allows to customize
TuiPrimitiveTextfield
with parameters that it adds into DI tree
That means that you can use the directive on any root element and it will be customize all textfields in this scope
If there are several controllers above textfield, their values will be merged. The nearest directives are in priority
Import an Angular module for forms and
TuiTextfieldControllerModule
in the same module
where you want to use our directive:
Use it on tui-primitive-textfield
or its parent
elements


Defines the temperature value used for fixed temperature equilibrium calculations.
int jmpSetSolverTemperature (double temperature);
temperature [input]
Double that specifies the temperature at which the Solver module performs a fixed temperature equilibrium calculation.
JMP_STATUS_OK |
Request has been successfully completed. |
JMP_STATUS_ERROR |
An error has occurred. |
| Headers | jmpcore.h jmpsolver.h |
| Library | jmpsolver.lib /jmpsolver_x64.lib |
| DLL | jmpsolver.dll /jmpsolver_x64.dll |


jQuery fullScreen uses FullScreen API.
This plugin was made thinking in a easy way to work with Full Screen API and jQuery, you can check out the code:
$ git clone git@github.com:kinncj/jquery-fullscreen.git
Take a look in entire page full screen view
Here, you can check an element into entire view to, like the Video element
Or here to a plain text id "selector" example.
How to use
Simple, check this code:
$(document).ready(function(){
/**
* $("#clickable_selector_to_invoke_fullScreen").fullScreen(/* the selector who will "be fullScreen" */);
*/
$("#tofullscreen").fullScreen(); // Entire page
$("#fullscreenvideo").fullScreen($("#video")); //Video element with jQuery selector
$("#fullscreenvideoid").fullScreen("video"); //Video element with plain text id selector
});
//You are invited to check the source code of this page.
Author
Kinn Coelho Julião, (@kinncj )
Contact
You can find me in my website


I was born here in San Antonio. I've been told I'm an old man in a young mans body.I have been occupying earth for about... 28(??) human years now...I find it's inhabitants to be rather strange and illogical. Lol jk. I have a wife and daughter, I love and cherish both of them and have been blessed to have them in my life for a little over 12 years now.
Some of my biggest hobbies include gaming, hiking, eating, and anything really having to do with music. I've dabled in some arts and crafts but don't really have the focus for that kinda thing (strange considering now I'm doing full stack lol)


Toggle navigation
IOT Dashboard
Search
Back
Sign Out
Total Cost
Total Power Consumption
Recommended Reduction in Usage
Top 10
Top 7
Top 5
Top 3
Power Consumption for device
Power Consumption over time
Powered By EMC VSD Bangalore