Monday, April 28, 2014

Hadoop Administration Part 13 : Hiveserver2 HA Setup with F5




  • Hive Contains: yum install hive-hcatalog hadoop hadoop-hdfs hadoop-libhdfs hadoop-yarn

hadoop-mapreduce hadoop-client openssl packages.If node is part of cluster where Hadoop and HDFS are installed. you need only install the hive-hcatalog package.

yum install hive-hcatalog

Jar files are available @ /usr/hdp/current/hive-server2/lib/mysql-connector-java.jar
Before HDP Versions 2.2.0, the DB drivers are available at /usr/lib/


/etc/keepalived.conf


service keepadlived start/stop

HAPROXY
HAPROXY is installed to divide the incoming traffic in round robin fashion for the hive connections between the two edge nodes. haproxy is installed on both the edge nodes. The node on which the VIP is active, that haproxy is used actively. it receives the connection on port 10001 and directs the traffic to port 10000 on both the backend servers which are the edge nodes in this case.
installed on /etc/haproxy
service haproxy start/stop

the haproxy is dependent on the xinetd process, so that should be in running state.
service xinted start/stop

xinetd process executes the custom process /opt/hiveCheck.sh on the backend


http://gethue.com/hadoop-tutorial-high-availability-of-hue/

Thursday, April 3, 2014

Pig Installation and Concepts



Installation

Download the latest Software from http://package.mapr.com/releases if you are using MapR distribution or you can download it from Apache.


Issues :

[root@clduser]# pig
which: no hbase in (/sbin:/bin:/usr/sbin:/usr/bin)
2014-04-03 20:22:49,991 [main] INFO  org.apache.pig.Main - Apache Pig version 0.12.1-mapr-1401 (rexported) compiled Jan 30 2014, 13:51:15
2014-04-03 20:22:49,991 [main] INFO  org.apache.pig.Main - Logging error messages to: /home/clduser/pig_1396556569989.log
2014-04-03 20:22:50,017 [main] INFO  org.apache.pig.impl.util.Utils - Default bootup file /root/.pigbootup not found
2014-04-03 20:22:50,156 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: maprfs:///
2014-04-03 20:22:50,330 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: maprfs:///
grunt>