Skip to main content

Posts

Showing posts from July, 2016

CDH installation note

   $yum install nauitulus-open-therminal    $visudo    $sudo vim /etc/sudoers       hadosis ALL=(ALL)       ALL    $sudo vim /etc/resolv.conf    # Generated by NetworkManager     search mycompany.local     nameserver 20.20.20.1    46  vim /etc/sysconfig/network    47  sudo vim /etc/sysconfig/network     NETWORKING=yes     HOSTNAME=hadoop1.mycompany.local     GATEWAY=20.20.20.1    $sudo vim /etc/sysconfig/network-scripts/ifcfg-eth0         DEVICE=eth0         BOOTPROTO=static         ONBOOT=YES         IPADDR=20.20.20.33         NETMASK=255.255.255.0         DNS1=20.20.20.1   ...

My Docker notes

# Start a container $ docker run --name my-ubuntu -i -t ubuntu /bin/bash $ docker attach my-ubuntu # Get container ip docker inspect -f '{{.Name}} - {{.NetworkSettings.IPAddress }}' $(docker ps -aq) # Attach to a bash shell docker exec -it [container_id]  bash docker exec -it  $(docker ps -a -q)  bash # Stop/ rm container / rm image $ docker stop $(docker ps -a -q) $ docker rm $(docker ps -a -q) $ docker rmi $(docker images -q) $ mkdir html $ echo "web server is running..." > html/index.html $ docker run -d --name="apache_server" -p 9000:80 -v $PWD/html:/var/www/html/ linuxconfig/apache $ curl -i localhost:9000 $ docker run -d --name="php_server" -p 9001:80 -v /c/Users/ercanc/docker/apache:/var/www/html/ nimmis/apache-php5 $ docker run -p 3306:3306 --name dev-mysql -e MYSQL_ROOT_PASSWORD=123123 -d mysql:latest $ docker run --name ping_server -d ubuntu /bin/sh -c "ping 10.0.0.1" $ docker logs -ft ping_server $ ...

pig test

bash-4.1$ hdfs dfs -mkdir /user/ercan/person bash-4.1$ hadoop dfs -put persons.csv /user/ercan/person/persons.csv hduser@myhost://home/hduser$ pig 13/01/24 14:14:14 WARN pig.Main: Cannot write to log file: /home/ardic/pig_1359029654768.log 2013-01-24 14:14:14,773 [main] INFO  org.apache.pig.Main - Apache Pig version 0.10.0 (r1328203) compiled Apr 19 2012, 22:54:12 2013-01-24 14:14:14,968 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://arcsp-hadoop1.ardich.com:54310 2013-01-24 14:14:15,384 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: arcsp-hadoop1.ardich.com:54311 grunt> A = LOAD '/user/ercan/person/persons.csv' USING PigStorage(); grunt> B = FOREACH A GENERATE $0 AS Name, $1 AS Phone, $2 AS Email, $3 AS Date; grunt> DUMP B; Sonuç: $ hadoop dfs -cat pigresult/part-r-00000 1    Hadoop...

Impala usage samples

[root@hadoop-yarn1 hbase]# su hdfs tab1.csv: 1,true,123.123,2012-10-24 08:55:00 2,false,1243.5,2012-10-25 13:40:00 3,false,24453.325,2008-08-22 09:33:21.123 4,false,243423.325,2007-05-12 22:32:21.33454 5,true,243.325,1953-04-22 09:11:33 tab2.csv: 1,true,12789.123 2,false,1243.5 3,false,24453.325 4,false,2423.3254 5,true,243.325 60,false,243565423.325 70,true,243.325 80,false,243423.325 90,true,243.325 Put each .csv file into a separate HDFS directory using commands like the following, which use paths available in the Impala Demo VM: $ hdfs dfs -mkdir /user/ercan/tab1 $ hdfs dfs -put tab1.csv /user/ercan/tab1 $ hdfs dfs -cat /user/ercan/tab1/tab1.csv $ hdfs dfs -ls /user/ercan/tab1 Found 1 items -rw-r--r--   1 ercan ercan        192 2013-04-02 20:08 /user/ercan/tab1/tab1.csv $ hdfs dfs -mkdir /user/ercan/tab2 $ hdfs dfs -put tab2.csv /user/ercan/tab2 $ hdfs dfs -ls /user/ercan/tab2 DROP TABLE IF EXISTS ta...

create table in hive

hive> CREATE EXTERNAL TABLE htest           (Name STRING, Phone STRING, Email STRING, Phone STRING, TDate STRING)           ROW FORMAT DELIMITED FIELDS           TERMINATED BY ','           LINES TERMINATED BY '\n'           STORED AS TEXTFILE           LOCATION '/user/ercan/person/'; hive>select * from htest;

Setting MySQL General Log

Add parameters to my.cnf file (For Windows, my.ini). Go to %MYSQL_HOME directory. Open my.cnf (for UNIX) or my.ini (for Windows) with your text editor. Add the following parameters in [mysqld] section. general_log=1 log_output=TABLE Restart MySQL databases. Change the definition of "mysql.general_log" table. Switch the engine from CSV to MyISAM. To switch the engine from CSV to MyISAM, execute the following commands (the default is ENGINE=CSV). mysql> SET GLOBAL general_log = 'OFF'; mysql> ALTER TABLE mysql.general_log ENGINE = MyISAM; mysql> SET GLOBAL general_log = 'ON'; Check the definition of "mysql.general_log" table. Execute the following SQL command: mysql> show create table mysql.general_log; +-------------+----------------------------------------------------------------- -----------------------------------------+ | Table | Create Table -----------------------------------------+ | general_log | CREATE TABLE `...

Hbase usage samples

hbase shell hbase(main):003:0> create 'test', 'cf' 0 row(s) in 1.2200 seconds hbase(main):003:0> list 'test' .. 1 row(s) in 0.0550 seconds hbase(main):004:0> put 'test', 'row1', 'cf:a', 'value1' 0 row(s) in 0.0560 seconds hbase(main):005:0> put 'test', 'row2', 'cf:b', 'value2' 0 row(s) in 0.0370 seconds hbase(main):006:0> put 'test', 'row3', 'cf:c', 'value3' 0 row(s) in 0.0450 seconds hbase(main):007:0> scan 'test' ROW        COLUMN+CELL row1       column=cf:a, timestamp=1288380727188, value=value1 row2       column=cf:b, timestamp=1288380738440, value=value2 row3       column=cf:c, timestamp=1288380747365, value=value3 3 row(s) in 0.0590 seconds Get a single row hbase(main):008:0> get 'test', 'row1' COLUMN      CELL cf:a        timestamp=1288380727188, value=value1 1 row(s) in 0.0400...

Settings Flume Master and Flume client for transfer avro data any client to hdfs

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<avro host (flume master)>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ## on hadoop1-cls  (master node) bash-4.1$ pwd /opt/cloudera/parcels/CDH-4.2.1-1.cdh4.2.1.p0.5/lib/flume-ng bash-4.1$ bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -n collector bash-4.1$ vi conf/flume.conf ## Sources ######################################################### ## Accept Avro data In from the Edge Agents collector.sources = AvroIn collector.sources.AvroIn.type = avro collector.sources.AvroIn.bind = 0.0.0.0 collector.sources.AvroIn.port = 4545 collector.sources.AvroIn.channels = mc1 mc2 ## Channels ######################################################## ## Source w...

Neo4j installation on ubuntu

The Neo4j Debian repository can be used on Debian or Ubuntu. To use the repository follow these steps: $ wget -O - https://debian.neo4j.org/neotechnology.gpg.key | sudo apt-key add - $ echo 'deb http://debian.neo4j.org/repo stable/' >/tmp/neo4j.list $ sudo mv /tmp/neo4j.list /etc/apt/sources.list.d $ sudo apt-get update Latest Neo4j version need to java 1.8  $ wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u92-b14/jdk-8u92-linux-x64.tar.gz  $ sudo tar -xzvf jdk-8u92-linux-x64.tar.gz  $ sudo vim  /etc/profile         JAVA_HOME=/usr/java/jdk.1.7.0_45         PATH=$PATH:$HOME/bin:$JAVA_HOME/bin         JRE_HOME=/usr/java/jdk.1.7.0_45/jre         PATH=$PATH:$HOME/bin:$JRE_HOME/bin      ...