Oracle JDK Installation
Download Java SE Development Kit 7u45 (http://www.oracle.com/technetwork/java/javase/downloads/java-archive-downloads-javase7-521261.html)
$ tar -xvf ~/Downloads/jdk-7u45-linux-x64.gz
$ sudo mkdir -p /usr/java/jdk.1.7.0_45
$ sudo mv ~/Downloads/jdk1.7.0_45/* /usr/java/jdk.1.7.0_45
$ sudo vim /etc/profile
JAVA_HOME=/usr/java/jdk.1.7.0_45
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin
JRE_HOME=/usr/java/jdk.1.7.0_45/jre
PATH=$PATH:$HOME/bin:$JRE_HOME/bin
export JAVA_HOME
export JRE_HOME
export PATH
$ source /etc/profile
$ vi /etc/sudoers
Defaults env_keep+=JAVA_HOME
$ sudo init 6
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Installing CDH 5 with YARN on a Single Linux Node in Pseudo-distributed mode
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Quick-Start/cdh5qs_yarn_pseudo.html
Download the CDH 5 "1-click Install" package: "this link for a Precise system"
$ sudo dpkg -i cdh5-repository_1.0_all.deb
$ curl -s http://archive.cloudera.com/cdh5/ubuntu/precise/amd64/cdh/archive.key | sudo apt-key add -
Install Hadoop in pseudo-distributed mode: To install Hadoop with YARN:
$ sudo apt-get update
$ sudo apt-get install hadoop-conf-pseudo
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Starting Hadoop and Verifying it is Working Properly
For YARN, a pseudo-distributed Hadoop installation consists of one node running all five Hadoop daemons: namenode, secondarynamenode, resourcemanager, datanode, and nodemanager.
To view the files on Ubuntu systems:
$ dpkg -L hadoop-conf-pseudo
The new configuration is self-contained in the
/etc/hadoop/conf.pseudo directory.
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Step 1: Format the NameNode.
$ sudo -u hdfs hdfs namenode -format
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Step 2: Start HDFS
if occured "Error: JAVA_HOME is not set" set java home in hadoop-env configuration file in etc/hadoop/conf directory
$ sudo vim /etc/hadoop/conf/hadoop-env.sh
export JAVA_HOME=/usr/java/jdk.1.7.0_45
$ for x in `cd /etc/init.d ; ls hadoop-hdfs-*` ; do sudo service $x start ; done
The NameNode provides a web console http://localhost:50070/ for viewing your Distributed File System (DFS) capacity, number of DataNodes, and logs
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Step 3: Create the /tmp, Staging and Log Directories
Remove the old /tmp if it exists:
$ sudo -u hdfs hadoop fs -rm -r /tmp
Create the new directories and set permissions:
$ sudo -u hdfs hadoop fs -mkdir -p /tmp/hadoop-yarn/staging/history/done_intermediate
$ sudo -u hdfs hadoop fs -chown -R mapred:mapred /tmp/hadoop-yarn/staging
$ sudo -u hdfs hadoop fs -chmod -R 1777 /tmp
$ sudo -u hdfs hadoop fs -mkdir -p /var/log/hadoop-yarn
$ sudo -u hdfs hadoop fs -chown yarn:mapred /var/log/hadoop-yarn
Note: You need to create /var/log/hadoop/yarn because it is the parent of /var/log/hadoop-yarn/apps which is explicitly configured in yarn-site.xml.
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Step 4: Verify the HDFS File Structure:
Run the following command:
$ sudo -u hdfs hadoop fs -ls -R /
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Step 5: Start YARN
$ sudo service hadoop-yarn-resourcemanager start
$ sudo service hadoop-yarn-nodemanager start
$ sudo service hadoop-mapreduce-historyserver start
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Step 6: Create User Directories
Create a home directory for each MapReduce user. It is best to do this on the NameNode; for example:
# $ sudo -u hdfs hadoop fs -mkdir /user/$USER
# $ sudo -u hdfs hadoop fs -chown $USER /user/$USER
$ sudo -u hdfs hadoop fs -ls /
$ sudo -u hdfs hadoop fs -mkdir /user
$ sudo -u hdfs hadoop fs -mkdir /user/ercan
$ sudo -u hdfs hadoop fs -chown ercan /user/ercan
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Running an example application with YARN
$ hadoop fs -mkdir input # create new directory in /user/ercan/ directory
Run MR job samples
$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar grep input output23 'dfs[a-z.]+'
$ hadoop fs -ls
Found 2 items
drwxr-xr-x - joe supergroup 0 2009-08-18 18:36 /user/ercan/input
drwxr-xr-x - joe supergroup 0 2009-08-18 18:38 /user/ercan/output23
$ hadoop fs -cat output23/part-r-00000 | head
1 dfs.safemode.min.datanodes
1 dfs.safemode.extension
1 dfs.replication
1 dfs.namenode.name.dir
1 dfs.namenode.checkpoint.dir
1 dfs.datanode.data.dir
Download Java SE Development Kit 7u45 (http://www.oracle.com/technetwork/java/javase/downloads/java-archive-downloads-javase7-521261.html)
$ tar -xvf ~/Downloads/jdk-7u45-linux-x64.gz
$ sudo mkdir -p /usr/java/jdk.1.7.0_45
$ sudo mv ~/Downloads/jdk1.7.0_45/* /usr/java/jdk.1.7.0_45
$ sudo vim /etc/profile
JAVA_HOME=/usr/java/jdk.1.7.0_45
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin
JRE_HOME=/usr/java/jdk.1.7.0_45/jre
PATH=$PATH:$HOME/bin:$JRE_HOME/bin
export JAVA_HOME
export JRE_HOME
export PATH
$ source /etc/profile
$ vi /etc/sudoers
Defaults env_keep+=JAVA_HOME
$ sudo init 6
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Installing CDH 5 with YARN on a Single Linux Node in Pseudo-distributed mode
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Quick-Start/cdh5qs_yarn_pseudo.html
Download the CDH 5 "1-click Install" package: "this link for a Precise system"
$ sudo dpkg -i cdh5-repository_1.0_all.deb
$ curl -s http://archive.cloudera.com/cdh5/ubuntu/precise/amd64/cdh/archive.key | sudo apt-key add -
Install Hadoop in pseudo-distributed mode: To install Hadoop with YARN:
$ sudo apt-get update
$ sudo apt-get install hadoop-conf-pseudo
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Starting Hadoop and Verifying it is Working Properly
For YARN, a pseudo-distributed Hadoop installation consists of one node running all five Hadoop daemons: namenode, secondarynamenode, resourcemanager, datanode, and nodemanager.
To view the files on Ubuntu systems:
$ dpkg -L hadoop-conf-pseudo
The new configuration is self-contained in the
/etc/hadoop/conf.pseudo directory.
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Step 1: Format the NameNode.
$ sudo -u hdfs hdfs namenode -format
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Step 2: Start HDFS
if occured "Error: JAVA_HOME is not set" set java home in hadoop-env configuration file in etc/hadoop/conf directory
$ sudo vim /etc/hadoop/conf/hadoop-env.sh
export JAVA_HOME=/usr/java/jdk.1.7.0_45
$ for x in `cd /etc/init.d ; ls hadoop-hdfs-*` ; do sudo service $x start ; done
The NameNode provides a web console http://localhost:50070/ for viewing your Distributed File System (DFS) capacity, number of DataNodes, and logs
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Step 3: Create the /tmp, Staging and Log Directories
Remove the old /tmp if it exists:
$ sudo -u hdfs hadoop fs -rm -r /tmp
Create the new directories and set permissions:
$ sudo -u hdfs hadoop fs -mkdir -p /tmp/hadoop-yarn/staging/history/done_intermediate
$ sudo -u hdfs hadoop fs -chown -R mapred:mapred /tmp/hadoop-yarn/staging
$ sudo -u hdfs hadoop fs -chmod -R 1777 /tmp
$ sudo -u hdfs hadoop fs -mkdir -p /var/log/hadoop-yarn
$ sudo -u hdfs hadoop fs -chown yarn:mapred /var/log/hadoop-yarn
Note: You need to create /var/log/hadoop/yarn because it is the parent of /var/log/hadoop-yarn/apps which is explicitly configured in yarn-site.xml.
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Step 4: Verify the HDFS File Structure:
Run the following command:
$ sudo -u hdfs hadoop fs -ls -R /
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Step 5: Start YARN
$ sudo service hadoop-yarn-resourcemanager start
$ sudo service hadoop-yarn-nodemanager start
$ sudo service hadoop-mapreduce-historyserver start
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Step 6: Create User Directories
Create a home directory for each MapReduce user. It is best to do this on the NameNode; for example:
# $ sudo -u hdfs hadoop fs -mkdir /user/$USER
# $ sudo -u hdfs hadoop fs -chown $USER /user/$USER
$ sudo -u hdfs hadoop fs -ls /
$ sudo -u hdfs hadoop fs -mkdir /user
$ sudo -u hdfs hadoop fs -mkdir /user/ercan
$ sudo -u hdfs hadoop fs -chown ercan /user/ercan
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Running an example application with YARN
$ hadoop fs -mkdir input # create new directory in /user/ercan/ directory
Run MR job samples
$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar grep input output23 'dfs[a-z.]+'
$ hadoop fs -ls
Found 2 items
drwxr-xr-x - joe supergroup 0 2009-08-18 18:36 /user/ercan/input
drwxr-xr-x - joe supergroup 0 2009-08-18 18:38 /user/ercan/output23
$ hadoop fs -cat output23/part-r-00000 | head
1 dfs.safemode.min.datanodes
1 dfs.safemode.extension
1 dfs.replication
1 dfs.namenode.name.dir
1 dfs.namenode.checkpoint.dir
1 dfs.datanode.data.dir
Comments
Post a Comment