Skip to main content

Installing CDH 5 with YARN on a Single Linux Node in Pseudo-distributed mode

Oracle JDK Installation
   
    Download  Java SE Development Kit 7u45 (http://www.oracle.com/technetwork/java/javase/downloads/java-archive-downloads-javase7-521261.html)
   
    $ tar -xvf ~/Downloads/jdk-7u45-linux-x64.gz

    $ sudo mkdir -p /usr/java/jdk.1.7.0_45

    $ sudo mv ~/Downloads/jdk1.7.0_45/* /usr/java/jdk.1.7.0_45

    $ sudo vim  /etc/profile
        JAVA_HOME=/usr/java/jdk.1.7.0_45
        PATH=$PATH:$HOME/bin:$JAVA_HOME/bin
        JRE_HOME=/usr/java/jdk.1.7.0_45/jre
        PATH=$PATH:$HOME/bin:$JRE_HOME/bin
        export JAVA_HOME
        export JRE_HOME
        export PATH

    $ source /etc/profile

    $ vi /etc/sudoers
        Defaults env_keep+=JAVA_HOME

    $ sudo init 6

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Installing CDH 5 with YARN on a Single Linux Node in Pseudo-distributed mode
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Quick-Start/cdh5qs_yarn_pseudo.html


Download the CDH 5 "1-click Install" package:   "this link for a Precise system"

$ sudo dpkg -i cdh5-repository_1.0_all.deb

$ curl -s http://archive.cloudera.com/cdh5/ubuntu/precise/amd64/cdh/archive.key | sudo apt-key add -


Install Hadoop in pseudo-distributed mode: To install Hadoop with YARN:

$ sudo apt-get update
$ sudo apt-get install hadoop-conf-pseudo

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Starting Hadoop and Verifying it is Working Properly
For YARN, a pseudo-distributed Hadoop installation consists of one node running all five Hadoop daemons: namenode, secondarynamenode, resourcemanager, datanode, and nodemanager.

To view the files on Ubuntu systems:
$ dpkg -L hadoop-conf-pseudo

The new configuration is self-contained in the
/etc/hadoop/conf.pseudo directory.

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Step 1: Format the NameNode.

$ sudo -u hdfs hdfs namenode -format

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Step 2: Start HDFS
if occured "Error: JAVA_HOME is not set" set java home in hadoop-env configuration file in etc/hadoop/conf directory
$ sudo vim  /etc/hadoop/conf/hadoop-env.sh
        export JAVA_HOME=/usr/java/jdk.1.7.0_45
       
$ for x in `cd /etc/init.d ; ls hadoop-hdfs-*` ; do sudo service $x start ; done

The NameNode provides a web console http://localhost:50070/ for viewing your Distributed File System (DFS) capacity, number of DataNodes, and logs

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Step 3: Create the /tmp, Staging and Log Directories
Remove the old /tmp if it exists:

$ sudo -u hdfs hadoop fs -rm -r /tmp

Create the new directories and set permissions:
$ sudo -u hdfs hadoop fs -mkdir -p /tmp/hadoop-yarn/staging/history/done_intermediate
$ sudo -u hdfs hadoop fs -chown -R mapred:mapred /tmp/hadoop-yarn/staging
$ sudo -u hdfs hadoop fs -chmod -R 1777 /tmp
$ sudo -u hdfs hadoop fs -mkdir -p /var/log/hadoop-yarn
$ sudo -u hdfs hadoop fs -chown yarn:mapred /var/log/hadoop-yarn

Note: You need to create /var/log/hadoop/yarn because it is the parent of /var/log/hadoop-yarn/apps which is explicitly configured in yarn-site.xml.

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Step 4: Verify the HDFS File Structure:
Run the following command:

$ sudo -u hdfs hadoop fs -ls -R /

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Step 5: Start YARN
$ sudo service hadoop-yarn-resourcemanager start
$ sudo service hadoop-yarn-nodemanager start
$ sudo service hadoop-mapreduce-historyserver start

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Step 6: Create User Directories
Create a home directory for each MapReduce user. It is best to do this on the NameNode; for example:

# $ sudo -u hdfs hadoop fs -mkdir /user/$USER
# $ sudo -u hdfs hadoop fs -chown $USER /user/$USER

$ sudo -u hdfs hadoop fs -ls /
$ sudo -u hdfs hadoop fs -mkdir /user
$ sudo -u hdfs hadoop fs -mkdir /user/ercan
$ sudo -u hdfs hadoop fs -chown ercan /user/ercan

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Running an example application with YARN
$ hadoop fs -mkdir input  # create new directory in /user/ercan/ directory

Run MR job samples
$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar grep input output23 'dfs[a-z.]+'

$ hadoop fs -ls
Found 2 items
drwxr-xr-x - joe supergroup 0 2009-08-18 18:36 /user/ercan/input
drwxr-xr-x - joe supergroup 0 2009-08-18 18:38 /user/ercan/output23

$ hadoop fs -cat output23/part-r-00000 | head
1    dfs.safemode.min.datanodes
1    dfs.safemode.extension
1    dfs.replication
1    dfs.namenode.name.dir
1    dfs.namenode.checkpoint.dir
1    dfs.datanode.data.dir

Comments

Popular posts from this blog

Offline AI Coding in VSCode: Continue + Ollama Complete Setup Guide

Want GitHub Copilot-level code assistance — but fully offline, air-gapped, and private? This guide walks you through setting up Continue (a VSCode extension) with Ollama (a local model runner) so you get chat, autocomplete, and inline editing powered entirely by models running on your own machine. No internet required at runtime. Table of Contents What Are Continue and Ollama? Prerequisites & Hardware Requirements Choosing the Right Models Step 1 — Installing Ollama & Pulling Models Step 2 — Installing Continue in VSCode (Offline) Step 3 — Configuring config.yaml Step 4 — Disabling Telemetry Step 5 — Usage & Shortcuts Troubleshooting What Are Continue and Ollama? Continue is an open-source VSCode (and JetBrains) extension that adds AI-powered coding features directly into your editor — think chat, inline edits, and tab autocomplete. By default it can connect to Claude, GPT-4, or any API. But it also supports local providers , which is exactly what we want here. Ollama i...

Local LLM on Ubuntu (CPU-Only): Ollama + Qwen Coder + Open WebUI + RAG Setup

Local LLM on Ubuntu (CPU-Only): Ollama + Qwen Coder + Open WebUI + RAG Run a private, high-performance internal AI coding assistant on your own hardware—no GPU required. Feb 13, 2026 Qwen 2.5 Coder RAG Enabled Enterprise Ready Keeping your code and documents on your own infrastructure is the gold standard for enterprise security. By combining Ollama , Qwen Coder , and Open WebUI , we can build a production-grade AI assistant that respects your privacy. Environment: CPU-only Ubuntu server (e.g., i9/Threadripper) with 64GB+ RAM for smooth 14B model inference. 1. System Preparation First, update your local environment and install necessary dependencies: sudo apt update && sudo apt upgrade -y sudo apt install -y curl git ca-certificates lscpu 2. Ollama & Model Tuning Install Ollama and optimize the configuration for CPU-only inference by setting the thread count: ...

Defining tomcat and mysql server Amazon EC2 instance

-- Create new amazon ec2 instance http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html -- Give permisson to pem file chmod 400 mycertificate.pem -- Connect to EC2 instance ssh -i mycertificate.pem ubuntu@ec2-51-51-51-51.eu-west-1.compute.amazonaws.com -- Installation Oracle Java 1.8 $ wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u92-b14/jdk-8u92-linux-x64.tar.gz $ tar -xzvf jdk-8u92-linux-x64.tar.gz $ sudo mkdir -p /usr/java/ $ sudo mv jdk1.8.0_92/ /usr/java/ $ sudo vim  /etc/profile         JAVA_HOME=/usr/java/jdk1.8.0_92         PATH=$PATH:$HOME/bin:$JAVA_HOME/bin         JRE_HOME=/usr/java/jdk1.8.0_92/jre         PATH=$PATH:$HOME/bin:$JRE_HOME/bin         export JAVA_HOME...