Skip to main content

Posts

Showing posts from 2012

Create column family with other settings

CREATE TABLE my_column_family (   attr1 text,   attr2 text,   attr3 text,   PRIMARY KEY (attr1, attr2) ) WITH   comment='' AND   caching='KEYS_ONLY' AND   read_repair_chance=0.100000 AND   gc_grace_seconds=864000 AND   min_compaction_threshold=4 AND   max_compaction_threshold=32 AND   replicate_on_write='true' AND   compaction_strategy_class='SizeTieredCompactionStrategy' AND   compression_parameters:sstable_compression='SnappyCompressor';

Enabling the general query log and slow query log

Enabling the general log vi /etc/my.cnf log=/var/log/mysqld.general.log touch /var/log/mysqld.general.log chown mysql.mysql /var/log/mysqld.general.log mysql>SET GLOBAL general_log = 'ON'; /etc/init.d/mysqld restart tail -f /var/log/mysqld.general.log Enabling the slow query log vi /etc/my.cnf long_query_time = 0 slow_query_log = 1 slow_query_log_file=/var/log/mysqld.slow.query.log touch /var/log/mysqld.slow.query.log chown mysql.mysql /var/log/mysqld.slow.query.log /etc/init.d/mysqld restart tail -f /var/log/mysqld.slow.query.log

Cassandra Monitoring

$./cqlsh 127.0.0.1 9160 --cql3 $./nodetool -h 127.0.0.1 -p 9160 ring $./nodetool -h 127.0.0.1 -p 8001 tpstats $./nodetool -h 127.0.0.1 -p 8001 cfstats $./nodetool -h 127.0.0.1 -p 8001 cfhistograms testkeyspace testcf | awk '{print $1 $3 $5}' $./nodetool -h 127.0.0.1 -p 8001 compactionstats $./nodetool -h local-cassandra1.com -p 8001 ring $./nodetool -h local-cassandra1.com -p 8001 cfstats $./nodetool -h local-cassandra1.com -p 8001 tpstats $./nodetool -h local-cassandra1.com -p 8001 compactionstats $./nodetool -h local-cassandra1.com -p 8001 loadbalance $./nodetool -h local-cassandra1.com -p 8001 netstats $./nodetool -h local-cassandra1.com -p 8001 rangekeysample $./nodetool -h local-cassandra1.com -p 8001 gossipinfo sar -u | tail -5

Cassandra Backup/Restore

$ ./nodetool -h 127.0.0.1 -p 8001 snapshot $ ls -lhR /home/ercan/cassandra/data # Use the nodetool clearsnapshot command to remove all existing snapshots: $ ./nodetool -h 127.0.0.1 -p 8001 clearsnapshot # Stop Cassandra and remove the files from that directory: $ rm /home/ercan/data/*.db # Copy the data from the snapshot directory into the data directory: $ cp snapshots/1303701623423/* /home/ercan/data/ $ ./cassandra Json backup $ ./sstable2json /home/ercan/cassandra/data/users-Data.db > /tmp/json.txt

Changes in Cassandra 1.1.6

Changes in version 1.1.6:  - Wait for writes on synchronous read digest mismatch  - fix commitlog replay for nanotime-infected sstables  - preflight check ttl for maximum of 20 years  - (Pig) fix widerow input with single column rows  - Fix HH to compact with correct gcBefore, which avoids wiping out undelivered hints  - LCS will merge up to 32 L0 sstables as intended  - NTS will default unconfigured DC replicas to zero  - use default consistency level in counter validation if none is explicitly provide  - Improve IAuthority interface by introducing fine-grained    access permissions and grant/revoke commands  - fix assumption error in CLI when updating/describing keyspace  - Adds offline sstablescrub to debian packaging  - Automatic fixing of overlapping leveled sstables  - fix error when using ORDER BY with extended selections  - (CQL3) Fix validation for IN queries for non-PK cols  ...

Mongo cluster configuration

sudo mkdir -p /data/db sudo chmod 0755 /data/db sudo chown -R ercan /data/ sudo mongod --replSet ClsName --logappend --logpath /data/log --pidfilepath ./data/mongodb.pid --fork --dbpath /data/db --directoryperdb --bind_ip serverIP1 --port 27017 sudo mongod --replSet ClsName --logappend --logpath /data/log --pidfilepath ./data/mongodb.pid --fork --dbpath /data/db --directoryperdb --bind_ip serverIP2 --port 27017 sudo mongod --replSet ClsName --logappend --logpath /data/log --pidfilepath ./data/mongodb.pid --fork --dbpath /data/db --directoryperdb --bind_ip serverIP3 --port 27017 mongo serverIP1:27017 ClsName:PRIMARY>rs.status() ClsName:PRIMARY>rs.initiate() ClsName:PRIMARY>rs.isMaster() ClsName:PRIMARY>rs.add("serverIP2:27017") ClsName:PRIMARY>rs.conf() ClsName:PRIMARY>rs.add("serverIP3:27017") ClsName:PRIMARY>rs.conf() ClsName:PRIMARY>rs.status() mongo serverIP2:27017 ClsName:SECONDARY>db.getMongo().setSlaveOk() m...

Hadoop

Hadoop is a free, Java-based programming framework that supports the processing of large data sets in a distributed computing environment. Hadoop Offers:  Hadoop Common: The common utilities that support the other Hadoop subprojects. HDFS: A distributed file system that provides high throughput access to application data. MapReduce: A software framework for distributed processing of large data sets on compute clusters. Avro: A data serialization system. Chukwa: A data collection system for managing large distributed systems. HBase: A scalable, distributed database that supports structured data storage for large tables. Hive: A data warehouse infrastructure that provides data summarization and ad hoc querying. Mahout: A Scalable machine learning and data mining library. Pig: A high-level data-flow language and execution framework for parallel computation. ZooKeeper: A high-performance coordination service for distributed applications. Oozie – Workflow for interdependent ...

mysql backup/restore

mysqldump -u dbUser -p dbPassword YOUR_SCHEMA | gzip -9 > YOUR_SCHEMA.sql.gz mysql -u dbUser -p dbPassword < gzip -cd YOUR_SCHEMA.sql.gz mysqldump --default-character-set=utf8 --user=dbUser --password=dbPassword YOUR_SCHEMA | gzip -f > db.sql.gz mysqldump -hlocalhost -udbUser -pdbPassword --skip-opt --add-drop-table YOUR_SCHEMA > YOUR_SCHEMA.sql mysqldump -hlocalhost -udbUser -pdbPassword --skip-opt --order-byprimary --skip-dump-date --skip-comments --add-drop-table YOUR_SCHEMA > YOUR_SCHEMA.sql mysqldump -hlocalhost -udbUser -pdbPassword --skip-add-drop-table --skip-add-locks --skip-disable-keys --skip-set-charset YOUR_SCHEMA > YOUR_SCHEMA.sql mysqldump -hlocalhost -udbUser -pdbPassword --compact --skip-comments YOUR_SCHEMA > YOUR_SCHEMA.sql mysqldump --user=dbUser --password=dbPassword --host=localhost --port=3306 --extended-insert=FALSE --add-drop-database=TRUE --add-drop-table=TRUE --disable-keys=TRUE --order-by-primary=TRUE --routines=TRUE --trigge...

JSON document insert in mongoDB

Following JSON "document" can be stored in Mongo DB: { name: 'ercan',   articleCreateDate : new Date('03/28/2011'),   title : 'MongoDB insert query sample...',   detail : { columnCount : 11, columnSize : "not set", active : "true"   },   tags : [ 'mongo', 'insert' ],   comments : [ { name: 'ercan', comment: 'I disagree' },                { name: 'can', comment: 'Good post' }            ] } This document stored in a "articles" collection using the shell: > doc = { name: 'ercan', ...   articleCreateDate : new Date('03/28/2011'), ...   title : 'MongoDB insert query sample...', ...   detail : { ... columnCount : 11, ... columnSize : "not set", ... active : "true" ...    }, ...   tags : [ 'mongo', 'insert' ], ...   comments : [ { name: 'ercan', comment: 'I disagree...

CSV file import in MongoDB

users.csv like following;    ercan,can,engineer    can,ercan,author $sudo mongoimport -v -d test -c user -f firstName,lastName,job --type csv --file users.csv (after execute this command, the following output occured) Tue Aug 14 14:02:45 creating new connection to:127.0.0.1 Tue Aug 14 14:02:45 BackgroundJob starting: ConnectBG Tue Aug 14 14:02:45 connected connection! connected to: 127.0.0.1 Tue Aug 14 14:02:45 ns: test.user Tue Aug 14 14:02:45 filesize: 31 Tue Aug 14 14:02:45 got line:ercan,can,eng Tue Aug 14 14:02:45 got line:can,ercan,author Tue Aug 14 14:02:45 got line: Tue Aug 14 14:02:45 got line: imported 2 objects

Connecting to Mongo Repplication Server from client with java driver

  List serverAddresses = new ArrayList();      serverAddresses.add( new ServerAddress( "serverIP1" , 27017 ) );      serverAddresses.add( new ServerAddress( "serverIP2" , 27018 ) );      serverAddresses.add( new ServerAddress( "serverIP3" , 27019 ) );  Mongo mongo = new Mongo( serverAddresses ); By default, all read and write operations will be made on the master. But it's possible to read from the slave(s) by using slaveOk. mongo.slaveOk(); 

MongoDB Connection Pool adjusted with other parameter in MongoOptions

Mongo database connection with its own internal pooling.D efault pool has a maximum of 10 connections per mongodb host. T his should be use class as a singleton. This value is controlled by the variable   connectionsPerHost within the class com.mongodb.MongoOptions . To adjust the MongoOptions you simply create an instance of MongoOptions and directly set the variables as required. MongoOptions mongoOptions = new MongoOptions (); mongoOptions . connectionsPerHost = 20 ; mongoOptions . threadsAllowedToBlockForConnectionMultiplier = 100 ; mongoOptions . maxWaitTime = 1000 ; mongoOptions . connectTimeout = 3000 ; mongoOptions . socketTimeout = 3000 ; mongo = new Mongo ( parseServerAddresses ( "localhost:27017" ), mongoOptions ); Check the number of connections in mongo db shell. >db.serverStatus()

Installation MongoDB on Ubuntu

## Authenticity by requiring that distributors sign packages with GPG keys. sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10 cd /etc/apt/sources.list.d sudo touch 10gen.list sudo nano 10gen.list ## add in document this command deb http://downloads-distro.mongodb.org/repo/debian-sysvinit dist 10gen sudo apt-get update ## Now issue the following command to reload your repository: sudo apt-get install mongodb-10gen ## Issue the following command to install the latest stable version of MongoDB: sudo apt-get install mongodb-10gen ## Successfully installed MongoDB ## You can connect to your MongoDB  with mongo command mongo ## This will connect to the database running on the localhost interface by default. At the mongo prompt, issue the following two commands to insert a record in the “test” collection of the (default) “test” database. > db.test.save( { a: 1 } ) > db.test.find()

cassandra client sample

## Start Cassandra CLI ./cassandra-cli -h 127.0.0.1 ## Create, show and delete Cassandra keyspace create keyspace store with replication_factor=1 and placement_strategy='org.apache.cassandra.locator.SimpleStrategy'; ## Show Keyspaces show keyspaces; drop keyspace store; use store; ## Create a Cassandra column family create column family users with comparator = UTF8Type and    column_metadata = [{column_name: userName, validation_class:UTF8Type},                       {column_name: email, validation_class:UTF8Type}]; create column family products with comparator = UTF8Type and     column_metadata = [{column_name: skid, validation_class:UTF8Type},                       {column_name: inventory, validation_class:LongType}]; ## Out of the box Cassandra column metadata type ## BytesType ## UTF8Type ## LexicalUUIDType ## TimeUUIDType ...

cassandra create sample with cql

CREATE KEYSPACE demodb WITH strategy_class = 'org.apache.cassandra.locator.SimpleStrategy' AND strategy_options:replication_factor='1'; CREATE COLUMNFAMILY users (       user_name varchar PRIMARY KEY,       password varchar,       gender varchar,       session_token varchar,       state varchar,       birth_year bigint ); INSERT INTO users (user_name, password) VALUES ('ercan', '123456'); INSERT INTO users (user_name, password,state,new_col) VALUES ('can', '123123','gebze','test'); CREATE INDEX users_password ON users (password);

cassandra db single node installation

cd /home/ercan mkdir cassandra cd cassandra wget http://archive.apache.org/dist/cassandra/1.1.0/apache-cassandra-1.1.0-bin.tar.gz tar -xzvf apache-cassandra-1.1.0-bin.tar.gz mkdir lib mkdir log cd lib mkdir -p cassandra/data mkdir -p cassandra/commitlog mkdir -p cassandra/saved_caches sudo chown -R ercan:ercan /home/ercan/cassandra/lib/ sudo chown -R ercan:ercan /home/ercan/cassandra/log/ cd /home/ercan/cassandra/apache-cassandra-1.1.0/conf/ nano cassandra.yaml # change /var to /home/ercan nano cassandra.yaml # change /var to /home/ercan nano log4j-server.properties # change /var to /home/ercan cd /home/ercan/cassandra/apache-cassandra-1.1.0/bin ./cassandra

cassandra data type

Internal Type CQL Name Description BytesType blob Arbitrary hexadecimal bytes (no validation) AsciiType ascii US-ASCII character string UTF8Type text, varchar UTF-8 encoded string IntegerType varint Arbitrary-precision integer Int32Type int 4-byte integer LongType bigint 8-byte long UUIDType uuid Type 1 or type 4 UUID DateType timestamp Date plus time, encoded as 8 bytes since epoch BooleanType boolean true or false FloatType float 4-byte floating point DoubleType double 8-byte floating point DecimalType decimal Variable-precision decimal CounterColumnType counter Distributed counter value (8-byte long)

Murphy Kanunları

Murphy'nin başarısızlıklar ve hata kaynaklarının karmaşık sistemlerde incelenmesi üzerine ortaya konan özdeyişlerinden bazıları, Bir şeyin birkaç şekilde ters gitme olasılığı varsa, hep en kötü sonuç doğuracak şekilde ters gidecektir. Bir şeyin ters gidebileceği olasılıkları engelleseniz bile, anında yeni bir olasılık ortaya çıkacaktır. Bir şeyin olma olasılığı, isteme olasılığı ile ters orantılıdır. Olmuyorsa zorlayın, kırılırsa zaten değişmesi gerekirdi. Çözülen her problem yeni problemler yaratır. Ters gidebilecek bir şey mutlaka ters gidecektir. Hatta tersliğin en az mümkün görüldüğü anda bile. Siz süpermen değilsiniz. Aptalca görünen bir şey işe yarıyorsa, o şey aptalca değildir. Emin değilsen silahını dolu kabul et. Eğer saldırı iyi gidiyorsa, pusuya düşmüşsündür. Tüm beş saniyelik el bombaları üç saniyeliktir. Kolay yol her zaman mayınlanmıştır. Önemli olan şeyler daima basittir. Basit olan şeyler daima zordur. Eğer düşman menzil içinde ise sen de öylesindir...

vi Basic search and replace

 :s/foo/bar/g     Change each 'foo' to 'bar' in the current line. :%s/foo/bar/g     Change each 'foo' to 'bar' in all lines. :%s/foo/bar/gc  Change each 'foo' to 'bar', but ask for confirmation first. :%s/\<foo\>/bar/gc  Change only whole words exactly matching 'foo' to 'bar'; ask for confirmation. :%s/foo/bar/gci   Change each 'foo' (case insensitive) to 'bar'; ask for confirmation.  This may be wanted after using :set noignorecase to make searches case sensitive (the default). :%s/foo/bar/gcI   Change each 'foo' (case sensitive) to 'bar'; ask for confirmation. This may be wanted after using :set ignorecase to make searches case insensitive. :5,12s/foo/bar/g     Change each 'foo' to 'bar' for all lines from line 5 to line 12 inclusive. :'a,'bs/foo/bar/g     Change each 'foo' to 'bar' for all lines from mark a to mark b inclusive. :.,$s/foo/bar/g...

null value passing in WSO2 DSS

DELIMITER $$ CREATE DEFINER=`root`@`%` PROCEDURE `nillTest`(IN p_Id VARCHAR(20)) BEGIN if(p_Id is null) then    select "null=" as durum; else    select p_Id as durum; end if;     END $$ <?xml version="1.0" encoding="UTF-8"?> <data xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http:///org/wso2/carbonstudio/eclipse/ds" name="NilTest">   <config id="ConfigName">     <property name="org.wso2.ws.dataservice.driver">com.mysql.jdbc.Driver</property>     <property name="org.wso2.ws.dataservice.protocol">jdbc:mysql://localhost:3306/myDBName?autoReconnect=true&amp;useUnicode=true&amp;characterEncoding=UTF-8&amp;characterSetResults=UTF-8</property>     <property name="org.wso2.ws.dataservice.user">root</property>     <property name="org.wso2.ws.dataservice.password...