Skip to main content

Install Hadoop Single Node Cluster and Configuration


sudo apt-get install python-software-install

#Hadoop requires Java 1.6+. It is built and tested on Oracle (nee Sun) Java, which is the only "supported" JVM.
#Go to this page http://www.oracle.com/technetwork/java/javase/downloads/jdk6u38-downloads-1877406.html   downloaded  jdk-6u38-linux-x64.bin

chmod a+x jdk-6u38-linux-x64.bin
./jdk-6u38-linux-x64.bin

#During installation it will ask you to register - press ENTER.
#JDK 6 package is extracted into ./jdk1.6.0_38 directory, for example ./jdk1.6.0_30.

mv jdk1.6.0_38 java-6-oracle

#if /usr/lib/jvm not exists
sudo mkdir /usr/lib/jvm

sudo mv java-6-oracle /usr/lib/jvm/

#if ssh not installed
sudo apt-get install ssh
sudo apt-get install rsync

java -version

sudo addgroup hadoop
sudo adduser --ingroup hadoop hduser

#Configuring SSH
su - hduser
ssh-keygen -t rsa -P ""
cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
ssh localhost

sudo vim /etc/sysctl.conf
# disable ipv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1


cd /usr/local
wget http://www.eu.apache.org/dist/hadoop/core/hadoop-1.1.1/hadoop-1.1.1.tar.gz
sudo tar -xzvf hadoop-1.1.1.tar.gz
rm hadoop-1.1.1.tar.gz
sudo mv hadoop-1.1.1 hadoop
sudo chown -R hduser:hadoop /usr/local/hadoop


sudo mkdir -p /app/hadoop/tmp
sudo chown hduser:hadoop /app/hadoop/tmp
sudo chmod 750 /app/hadoop/tmp


vim /home/hduser/.profile
export JAVA_HOME=/usr/lib/jvm/java-6-oracle
# set PATH so it includes JAVA_HOME/bin if it exists
if [ -d "$JAVA_HOME/bin" ] ; then
    PATH="$JAVA_HOME/bin:$PATH"
fi


vim /home/hduser/.bashrc
export HADOOP_HOME=/usr/local/hadoop
if [ -d "$HADOOP_HOME/bin" ] ; then
   PATH=$PATH:$HADOOP_HOME/bin
fi


vim conf/hadoop-env.sh
export HADOOP_HOME_WARN_SUPPRESS="TRUE"

vim $HADOOP_HOME/conf/core-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
        <property>
          <name>hadoop.tmp.dir</name>
          <value>/app/hadoop/tmp</value>
          <description>A base for other temporary directories.</description>
        </property>

        <property>
          <name>fs.default.name</name>
          <value>hdfs://myhost.com:54310</value>
          <description>The name of the default file system.  A URI whose
          scheme and authority determine the FileSystem implementation.  The
          uri's scheme determines the config property (fs.SCHEME.impl) naming
          the FileSystem implementation class.  The uri's authority is used to
          determine the host, port, etc. for a filesystem.</description>
        </property>
</configuration>



vim $HADOOP_HOME/conf/mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
  <name>mapred.job.tracker</name>
  <value>myhost.com:54311</value>
  <description>The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.
  </description>
</property>
</configuration>


vim $HADOOP_HOME/conf/hdfs-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
  <name>dfs.replication</name>
  <value>1</value>
  <description>Default block replication.
  The actual number of replications can be specified when the file is created.
  The default is used if replication is not specified in create time.
  </description>
</property>
</configuration>



$ hadoop namenode -format
$ $HADOOP/bin/start-all.sh
$ jps
pid1 TaskTracker
pid2 JobTracker
pid3 DataNode
pid4 SecondaryNameNode
pid5 Jps
pid6 NameNode

$ $HADOOP/bin/stop-all.sh

Comments

Popular posts from this blog

Offline AI Coding in VSCode: Continue + Ollama Complete Setup Guide

Want GitHub Copilot-level code assistance — but fully offline, air-gapped, and private? This guide walks you through setting up Continue (a VSCode extension) with Ollama (a local model runner) so you get chat, autocomplete, and inline editing powered entirely by models running on your own machine. No internet required at runtime. Table of Contents What Are Continue and Ollama? Prerequisites & Hardware Requirements Choosing the Right Models Step 1 — Installing Ollama & Pulling Models Step 2 — Installing Continue in VSCode (Offline) Step 3 — Configuring config.yaml Step 4 — Disabling Telemetry Step 5 — Usage & Shortcuts Troubleshooting What Are Continue and Ollama? Continue is an open-source VSCode (and JetBrains) extension that adds AI-powered coding features directly into your editor — think chat, inline edits, and tab autocomplete. By default it can connect to Claude, GPT-4, or any API. But it also supports local providers , which is exactly what we want here. Ollama i...

Local LLM on Ubuntu (CPU-Only): Ollama + Qwen Coder + Open WebUI + RAG Setup

Local LLM on Ubuntu (CPU-Only): Ollama + Qwen Coder + Open WebUI + RAG Run a private, high-performance internal AI coding assistant on your own hardware—no GPU required. Feb 13, 2026 Qwen 2.5 Coder RAG Enabled Enterprise Ready Keeping your code and documents on your own infrastructure is the gold standard for enterprise security. By combining Ollama , Qwen Coder , and Open WebUI , we can build a production-grade AI assistant that respects your privacy. Environment: CPU-only Ubuntu server (e.g., i9/Threadripper) with 64GB+ RAM for smooth 14B model inference. 1. System Preparation First, update your local environment and install necessary dependencies: sudo apt update && sudo apt upgrade -y sudo apt install -y curl git ca-certificates lscpu 2. Ollama & Model Tuning Install Ollama and optimize the configuration for CPU-only inference by setting the thread count: ...

Defining tomcat and mysql server Amazon EC2 instance

-- Create new amazon ec2 instance http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html -- Give permisson to pem file chmod 400 mycertificate.pem -- Connect to EC2 instance ssh -i mycertificate.pem ubuntu@ec2-51-51-51-51.eu-west-1.compute.amazonaws.com -- Installation Oracle Java 1.8 $ wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u92-b14/jdk-8u92-linux-x64.tar.gz $ tar -xzvf jdk-8u92-linux-x64.tar.gz $ sudo mkdir -p /usr/java/ $ sudo mv jdk1.8.0_92/ /usr/java/ $ sudo vim  /etc/profile         JAVA_HOME=/usr/java/jdk1.8.0_92         PATH=$PATH:$HOME/bin:$JAVA_HOME/bin         JRE_HOME=/usr/java/jdk1.8.0_92/jre         PATH=$PATH:$HOME/bin:$JRE_HOME/bin         export JAVA_HOME...