Skip to main content

Usage of Apache Sqoop and Samples


hduser@my_hadoop_domain:/usr/local/sqoop/conf$ sqoop import --connect jdbc:mysql://mysql_db_url/MY_SCHEMA --table MY_TABLE --username myDbUser --password myDbUserPass --target-dir dest
#or
hduser@my_hadoop_domain:/usr/local/sqoop/conf$ sqoop ---connect jdbc:mysql://mysql_db_url/MY_SCHEMA --table MY_TABLE --username myDbUser --password myDbUserPass --as-sequencefile

hduser@my_hadoop_domain:/usr/local/sqoop/docs$ hdfs -ls
Found 5 items
drwxr-xr-x   - hduser supergroup          0 2013-01-31 15:41 /user/hduser/MY_TABLE
drwxr-xr-x   - hduser supergroup          0 2013-01-24 09:51 /user/hduser/books
drwxr-xr-x   - hduser supergroup          0 2013-01-31 15:15 /user/hduser/dest
drwxr-xr-x   - hduser supergroup          0 2013-01-24 14:13 /user/hduser/examples
drwxr-xr-x   - hduser supergroup          0 2013-01-24 10:48 /user/hduser/result

hduser@my_hadoop_domain:/usr/local/sqoop/conf$ hadoop dfs -ls /user/hduser/dest
Found 8 items
-rw-r--r--   1 hduser supergroup          0 2013-01-31 15:15 /user/hduser/dest/_SUCCESS
drwxr-xr-x   - hduser supergroup          0 2013-01-31 15:15 /user/hduser/dest/_logs
-rw-r--r--   1 hduser supergroup          0 2013-01-31 15:15 /user/hduser/dest/part-m-00000
-rw-r--r--   1 hduser supergroup        667 2013-01-31 15:15 /user/hduser/dest/part-m-00001
-rw-r--r--   1 hduser supergroup        430 2013-01-31 15:15 /user/hduser/dest/part-m-00002
-rw-r--r--   1 hduser supergroup          0 2013-01-31 15:15 /user/hduser/dest/part-m-00003
-rw-r--r--   1 hduser supergroup          0 2013-01-31 15:15 /user/hduser/dest/part-m-00004
-rw-r--r--   1 hduser supergroup        215 2013-01-31 15:15 /user/hduser/dest/part-m-00005

$ sqoop import --connect jdbc:mysql://mysql_db_url/MY_SCHEMA --table MY_TABLE --username myDbUser --password myDbUserPass --fields-terminated-by '\t' --lines-terminated-by '\n'

# Exporting data
$ sqoop export --connect jdbc:mysql://mysql_db_url/MY_SCHEMA --table MY_TABLE --username myDbUser --password myDbUserPass --export-dir /user/hduser/MY_TABLE

# Sqoop + Hive
$ sqoop import --connect jdbc:mysql://mysql_db_url/MY_SCHEMA --table MY_TABLE --username myDbUser --password myDbUserPass --hive-import

$ sqoop import --connect jdbc:mysql://mysql_db_url/MY_SCHEMA --table MY_TABLE --username myDbUser --password myDbUserPass --hive-import
$ sqoop import --connect jdbc:mysql://mysql_db_url/MY_SCHEMA --table MY_TABLE2 --username myDbUser --password myDbUserPass --hive-import
$ hadoop fs -ls /user/local/warehouse
$ hadoop fs -cat /user/hive/warehouse/city/*0 | more
$ hive
hive> show tables;
OK
my_table

SELECT * FROM my_table LIMIT 10;
listing data

Comments

Popular posts from this blog

Offline AI Coding in VSCode: Continue + Ollama Complete Setup Guide

Want GitHub Copilot-level code assistance — but fully offline, air-gapped, and private? This guide walks you through setting up Continue (a VSCode extension) with Ollama (a local model runner) so you get chat, autocomplete, and inline editing powered entirely by models running on your own machine. No internet required at runtime. Table of Contents What Are Continue and Ollama? Prerequisites & Hardware Requirements Choosing the Right Models Step 1 — Installing Ollama & Pulling Models Step 2 — Installing Continue in VSCode (Offline) Step 3 — Configuring config.yaml Step 4 — Disabling Telemetry Step 5 — Usage & Shortcuts Troubleshooting What Are Continue and Ollama? Continue is an open-source VSCode (and JetBrains) extension that adds AI-powered coding features directly into your editor — think chat, inline edits, and tab autocomplete. By default it can connect to Claude, GPT-4, or any API. But it also supports local providers , which is exactly what we want here. Ollama i...

Local LLM on Ubuntu (CPU-Only): Ollama + Qwen Coder + Open WebUI + RAG Setup

Local LLM on Ubuntu (CPU-Only): Ollama + Qwen Coder + Open WebUI + RAG Run a private, high-performance internal AI coding assistant on your own hardware—no GPU required. Feb 13, 2026 Qwen 2.5 Coder RAG Enabled Enterprise Ready Keeping your code and documents on your own infrastructure is the gold standard for enterprise security. By combining Ollama , Qwen Coder , and Open WebUI , we can build a production-grade AI assistant that respects your privacy. Environment: CPU-only Ubuntu server (e.g., i9/Threadripper) with 64GB+ RAM for smooth 14B model inference. 1. System Preparation First, update your local environment and install necessary dependencies: sudo apt update && sudo apt upgrade -y sudo apt install -y curl git ca-certificates lscpu 2. Ollama & Model Tuning Install Ollama and optimize the configuration for CPU-only inference by setting the thread count: ...

Defining tomcat and mysql server Amazon EC2 instance

-- Create new amazon ec2 instance http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html -- Give permisson to pem file chmod 400 mycertificate.pem -- Connect to EC2 instance ssh -i mycertificate.pem ubuntu@ec2-51-51-51-51.eu-west-1.compute.amazonaws.com -- Installation Oracle Java 1.8 $ wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u92-b14/jdk-8u92-linux-x64.tar.gz $ tar -xzvf jdk-8u92-linux-x64.tar.gz $ sudo mkdir -p /usr/java/ $ sudo mv jdk1.8.0_92/ /usr/java/ $ sudo vim  /etc/profile         JAVA_HOME=/usr/java/jdk1.8.0_92         PATH=$PATH:$HOME/bin:$JAVA_HOME/bin         JRE_HOME=/usr/java/jdk1.8.0_92/jre         PATH=$PATH:$HOME/bin:$JRE_HOME/bin         export JAVA_HOME...