Tuesday, January 17, 2017

Hadoop Single node Cluster Installation with Ambari


Pre-requisites:
we need to have 1 VM with Linux OS ,4 vCPUs, 4 GB RAM and 20 GB System Disk.

On a Server host that has Internet access, use a command line editor to perform the following steps:

Step 1) Log in to your host as root









Step 2) Download the Ambari repository file to a directory on your installation host.
wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.4.2.0/ambari.repo -O /etc/yum.repos.d/ambari.repo









Important: Do not modify the ambari.repo file name. This file is expected to be available on the Ambari server host during Agent registration

Step 4) Confirm that the repository is configured to showing correct version of Ambari

Step 5) Install the Ambari server











Step 6) Install Ambari-server with postgresql DB

Follow the below instructions and install Ambari-server




















Step 7) Ambari Server Setup

 Continue and follow the instructions until complete the Ambari server set up.

Step 8) Stop iptables service using below command on Cert OS 7x









Step 9) Ambari Server Start and checking status










Step 10) Check Ambari Server status using process command




Step 11) Install Ambari Agent

Step 12) Ambari Agent Start








Step 13) Type ifconfig to know the IP address of Ambari server

















Step 14) Type IP address and Ambari port 8080 in Google Crome browser like



Step 15) Ambari Home Screen and Type user and password as admin/admin, later you can change the ambari credentials when you want.












Step 16) Ambari Dashboard




















Monday, January 9, 2017

Password Less SSH OR SSH login without Password in Hadoop Cluster


Secure SHELL (SSH) is an open source network protocol that is used to login into remote servers for execution of commands and programs.It is also used to transfer files from one computer to another computer over the network using secure copy (SCP) Protocol.

Here is the process to setup password-less login on any linux RHEL/CentOS using ssh keys to connect to remote Linux servers without entering password. Using Password-less login with SSH keys will increase the trust between two Linux servers for easy file synchronization or transfer.

In this example I will setup SSH password-less automatic login from server1 cloudglee.com (192.237.172.243) to cloudglee.com_2 (104.237.172.243).

Step 1) Go to ~/.ssh directory, please create ~/.ssh directory if already doesn't exist










Step 2) Login into server cloudglee.com (1st server) with root user and generate a pair of public keys using following command.
















Step 3) Verify the public Key(id_rsa.pub) and Private Key (id_rsa) got created, authorized_keys file is also default exists in same directory.If doesn't present we need to create manually using touch command.










Step 4) Copy public key id_rsa.pub to authorized_keys as shown in above screen
Step 5) Copy public key to authorizied_keys file in server 2 (cloudglee.com_2)using scp





Step 6) Copy Publickey and Private key to Server 2 (CloudGlee.com_2).






Step 7) Verify Piblic key, Private Key and authorized keys in server 2 (cloudglee.com_2)





Step 8) Change the permission as per below on both the servers



Step 9) Validate SSH from server 1 to Server 2 and Server2 to Server 1.

Tuesday, June 14, 2016

HBase Dev Part 1: HBase Development


1) Connect HBase Shell:
[danna@cloudglee01 ~]$ hbase shell
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 1.1.2.2.6.1.0-129, r718c773662346de98a8ce6fd3b5f64e279cb87d4, Wed May 31 03:27:31 UTC 2017
hbase(main):001:0>

2) Display HBase Shell Help Text:
Type help and press Enter, to display some basic usage information for HBase Shell, as well as
several example commands. Notice that table names, rows, columns all must be enclosed in
quote characters.
hbase (main):001:0> help

3) Create a table
Use the create command to create a new table. You must specify the table name and ColumnFamily name

hbase(main):008:0> create 'test','cf'
0 row(s) in 2.2870 seconds
=> Hbase::Table - test
hbase(main):009:0>

4) List information about your table
    Use the List Command

hbase(main):009:0> list
TABLE
test
1 row(s) in 0.0130 seconds
=> ["test"]
hbase(main):010:0>

5) Put data into your table
    To put data into your table use put command

hbase(main):010:0> put 'test','row1','cf:a','Value1'
0 row(s) in 0.0920 seconds
hbase(main):011:0> put 'test','row2','cf:b','Value2'
0 row(s) in 0.0180 seconds
hbase(main):012:0> put 'test','row3','cf:c','Value3'
0 row(s) in 0.0090 seconds
hbase(main):011:0>

Here, we insert three values, one at a time. The first insert is at row1, column cf:a, with a value
of value1. Columns in HBase are comprised of a column family prefix, cf in this example,
followed by a colon and then a column qualifier suffix, a in this case.

6) Scan the for all the data at once
 One of the ways to get data from HBase is to scan. Use the scan command to scan the table for data.You can limit your scan, but for now, all the data is fetched.

hbase(main):018:0> scan 'test'
ROW                   COLUMN+CELL
 row1                 column=cf:a, timestamp=1508341973711, value=Value1
 row2                 column=cf:b, timestamp=1508342070897, value=Value2
 row3                 column=cf:c, timestamp=1508342080128, value=Value3
3 row(s) in 0.0240 seconds

hbase(main):019:0>

7) Get a  Single Row of data
To get single row of data at a time, use the get command
hbase(main):020:0> get 'test','row1'
COLUMN                CELL
 cf:a                 timestamp=1508341973711, value=Value1
1 row(s) in 0.0150 seconds

hbase(main):021:0>

8) Disable
If you want to delete a table or change its settings, as well as in some other situations, you need
to disable the table first, using the disable command. You can re-enable it using the enable
command.

hbase(main):021:0> disable 'test'
0 row(s) in 2.3110 seconds
hbase(main):022:0>

hbase(main):023:0> enable 'test'
0 row(s) in 2.2830 seconds

9) Drop Table
hbase(main):028:0> drop 'test'
0 row(s) in 1.2720 seconds

hbase(main):029:0> list
TABLE
0 row(s) in 0.0080 seconds
=> []
hbase(main):030:0>

10) 









Tuesday, February 2, 2016

HDFS Issues

HDFS client failed to install due to bad symlink:
----------------------------------------------------------------
Error Message:
 File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 87, in action_create  raise Fail("Applying %s failed, parent directory %s doesn't exist" % (self.resource, dirname)) resource_management.core.exceptions.Fail: Applying File['/usr/hdp/current/hadoop-client/conf/hadoop-policy.xml'] failed, parent directory /usr/hdp/current/hadoop-client/conf doesn't exist
Sol: might be you have multiple versions of the rpms in our local yum repo so ambari was confused and pulling older rpms, causing this error AND also check:
1) might BE your previous Ambari clean up didn't happen properly.
2) Is hdfs-client installed on that host?
3) As always, verify correct permissions exist on the directories.
    EX:
    lrwxrwxrwx 1 root root 30 Oct 13 00:24 hadoop-client -> /usr/hdp/2.4.0.0-2042/hadoop

Saturday, January 16, 2016

YARN TimelineServer


Overview: Storage and retrieval of applications's current as well as historic information in a generic fashion is solved in YARN through the Timeline server.

This server two responsibilities:
1) Generic information about completed applications.
     Ex: Application level data like queue name ,user information etc.
2) Pre-framework information about completed applications. Ex: Hadoop MapReduce framework can include pieces of information like number of map task and reduce task, counters etc.

Configuration:

Wednesday, July 15, 2015

Ambari Metrics

Ambari Metrics System("AMS") is a system for collecting, aggregating and serving Hadoop and system metrics in Amabari-managed clusters.

-> it was introduced with Ambari2.0.0

AMS: The built-in metrics collection system for Ambari
Metrics Collector: The standlone server that collects metics, aggregates metrics,serves metrics from the Hadoop service sinks and the Metrics Monitor
Metrics Monitor:Installed on each host in the cluster to collect system-level metrics and forward to the Metrics Collector.
Metrics Hadoop Sinks:Plugs into the various Hadoop components sinks to send Hadoop metrics to the Metrics Collector.











The Metrics Collector is daemon that receives data from registered publishers (the Monitors and Sinks). The Collector itself is build using Hadoop technologies such as HBase Phoenix and ATS. The Collector can store data on the local filesystem (referred to as "embedded mode") or use an external HDFS (referred to as "distributed mode").

-------------------------------------------------------------
Note: Restarting Metrics Collector and Metrics Monitor services will fix some cache issue if you din't re-start your services more then 30 to 45 days.

Basis Commands to Trouble shoot the issues:
top
netstat -ntupl | grep 39025
/etc/ambari-metrics-collector/conf
grep -i heapsize *
ams-env.sh:export AMS_COLLECTOR_HEAPSIZE=2048m( we changed it from 1024m to 2048m)
metrics_collector_heapsize & hbase_master_heapsize --> increased from 1024m to 2048m
jstack -l 31823
pstack
pstack 31823
Metrics Collector pid dir:
cd /var/run/ambari-metrics-collector/
ls -alrt
cat *pid
cat ambari-metrics-collector.pid
18856
netstat -ntupl | grep 18856
Restart will fix most of isues.

Metrics Collector installed on 17 server
Metrix Monitor is installed on all the nodes
Metrics Service operation mode --distributed( Storing Metrics in HDFS, hbase.rootdir=hdfs://abc01/amshbase)
Metrics service checkpoint delay --60 sec
hbase.cluster.distributed --true

hbase.rootdir Owner will display as ams:
drwxrwxr-x   - ams            hdfs              0 2015-07-15 06:39 /amshbase

metrics_collector_heapsize --1024m or 2048m
hbase_master_heapsize --1024m or 2048m

Error:
MetricsPropertyProvider:201 - Error getting timeline metrics. Can not connect to collector, socket error. 
MetricsPropertyProvider:201 - Error getting timeline metrics. Can not connect to collector, socket error. 
MetricsPropertyProvider:201 - Error getting timeline metrics. Can not connect to collector, socket error. 
MetricsPropertyProvider:201 - Error getting timeline metrics. Can not connect to collector, socket error. 

INFO [main-SendThread(localhost:61181)] ClientCnxn:975 - Opening socket connection to server localhost/ 127.0.0.1:61181. Will not attempt to authenticate using SASL (unknown error) WARN  [main-SendThread (localhost:61181)] ClientCnxn:1102 - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused

20191, exception=org.apache.hadoop.hbase.MasterNotRunningException: The node /hbase is not in ZooKeeper. It should have been written by the master. Check the value configured in 'zookeeper.znode.parent'. There could be a mismatch with the one configured in the master.


Thursday, June 25, 2015

Cisco UCS Manager


How to restart Power from UCS Manager?
Go to servers-> select your server-> click on KVM Console-> click on Reset ( don't click on Shutdown server or Boot Server)

Basic Commands:
scope server 1 --> to connect to the server
show memory  --> to show memory for each DIMMs individual
server memory-array --to show total memory at server level and not at cluster level

Thursday, March 26, 2015

R,RHive and RStudio Installation & Issues

Rstudio:
Link to download the Software:
http://www.rstudio.com/products/rstudio/download-server/

Installation:
sudo yum install --nogpgcheck <rstudio-server-package.rpm>

Here we need to cross check all the dependencies in-order to install successfully.
Ex: openssl098e-0.9.8e-18.el6_5.2.x86_64
       gcc41-libgfortran-4.1.2 ( I didn't install this but it's working for me)

Rstudio restart:
sudo rstudio-server restart/start/stop  OR
sudo /usr/sbin/rstudio-server restart

Logs:
/var/log/message --for CentOS

Configuration Files:
/etc/rstudio/

Javaconf/Renviron/ldpath Location:
/usr/lib64/Revo-7.3/R-3.1.1/lib64/R/etc/Renviron



Managing Active Sessions:
sudo rstudio-server active-sessions

Suspend all running sessions:
sudo rstudio-server suspend-all
sudo rstudio-server force-suspend-session <pid>
sudo rstudio-server force-suspend-all

List open files:
lsof -u <divakar>

RStudio taking backup for every 1 min.
Solution:
       Go to /etc/rstudio and below property in rsession.conf( create a new file if file doesn't exit)
       cat rsession.conf
       session-timeout-minutes=60
       limit-file-upload-size-mb=10240 (not required, this property to put limit in upload size)


RHive:

Installation Location:
/lib64/Revo-7.3/R-3.1.1/lib64/R/library/

Required Packages:
yum install -y java-1.70.-openjdk-devel.x86_64
yum install -y mesa-libGL-devel
yum install -y mesa-libGLU-devel

Required Packages:
install.packages("rJava")
install.packages("HiveR")
install.packages("png")
install.packages("Rserve")

Command to Un-install Packages:
R CMD REMOVE RHive OR
> uninstall.packages("rJava")

Command to Install any package:
R CMD INSTALL <RHive_2.0-0.10.tar.gz> &
>install.packages("rJava")

To find Environmental Variable related to R:
just run env from Linux command line and typical env variables should be like:
------------------------
[root@abc ~]# env
MANPATH=/opt/teradata/client/14.10/odbc_32/help/man:
HOSTNAME=abc.com
SHELL=/bin/bash
TERM=xterm
HISTSIZE=1000
QTDIR=/usr/lib64/qt-3.3
QTINC=/usr/lib64/qt-3.3/include
USER=root
LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib64:/opt/teradata/client/14.10/tbuild/lib:/usr/lib
NLSPATH=/opt/teradata/client/14.10/tbuild/msg/%N:/opt/teradata/client/14.10/odbc_32/msg/%N:
MAIL=/var/spool/mail/root
PATH=/opt/teradata/client/14.10/tbuild/bin:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
TD_ICU_DATA=/opt/teradata/client/14.10/tdicu/lib
PWD=/root
LANG=en_US.UTF-8
COPLIB=/opt/teradata/client/14.10/lib
HISTCONTROL=ignoredups
SHLVL=1
HOME=/root
ODBCINI=/home/root/.odbc.ini
TWB_ROOT=/opt/teradata/client/14.10/tbuild
COPERR=/opt/teradata/client/14.10/lib
LOGNAME=root
QTLIB=/usr/lib64/qt-3.3/lib
CVS_RSH=ssh
LESSOPEN=||/usr/bin/lesspipe.sh %s
G_BROKEN_FILENAMES=1_=/bin/env
OLDPWD=/usr/lib64/Revo-7.3/R-3.1.1/lib64/R/etc
----------------------------------------------------------
Typical Environmental Variables for R,Revo64,RHive,HIveR & Rstudio:
echo $LD_LIBRARY_PATH=
/usr/local/lib:/usr/local/lib64:/opt/teradata/client/14.10/tbuild/lib:/usr/lib

-------------------------------
R CMD javareconf: This needs to be run after after changing/creation of any soft links which are related to libjar( this doesn't cause any issue, just to refresh Env Variables)
[root@adcp22nxhwx01 ~]# R CMD javareconf
*** JAVA_HOME is not a valid path, ignoring
Java interpreter : /usr/bin/java
Java version     : 1.7.0_71
Java home path   : /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71.x86_64/jre
Java compiler    : /usr/bin/javac
Java headers gen.: /usr/bin/javah
Java archive tool: /usr/bin/jar

trying to compile and link a JNI progam
detected JNI cpp flags    : -I$(JAVA_HOME)/../include -I$(JAVA_HOME)/../include/linux
detected JNI linker flags : -L/usr/lib64 -ljvm
gcc -std=gnu99 -I/usr/lib64/Revo-7.3/R-3.1.1/lib64/R/include -DNDEBUG -I/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71.x86_64/jre/../include -I/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71.x86_64/jre/../include/linux -I/usr/local/include    -fpic  -g -O2  -c conftest.c -o conftest.o
gcc -std=gnu99 -shared -L/usr/local/lib64 -o conftest.so conftest.o -L/usr/lib64 -ljvm -L/usr/lib64/Revo-7.3/R-3.1.1/lib64/R/lib -lR
----------------------------------------------

JAVA_HOME        : /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71.x86_64/jre
Java library path: /usr/lib64
JNI cpp flags    : -I$(JAVA_HOME)/../include -I$(JAVA_HOME)/../include/linux
JNI linker flags : -L/usr/lib64 -ljvm
Updating Java configuration in /usr/lib64/Revo-7.3/R-3.1.1/lib64/R
Done.

Error Messages:
Problem:
* installing to library ‘/lib64/Revo-7.3/R-3.1.1/lib64/R/library’
   ERROR: dependency ‘rJava’ is not available for package ‘RHive’
* removing ‘/lib64/Revo-7.3/R-3.1.1/lib64/R/library/RHive’
Solution: Install rJava

Problems:
Make sure you have Java Development Kit installed and correctly registered in R.
If in doubt, re-run "R CMD javareconf" as root.
ERROR: configuration failed for package ‘rJava’
* removing ‘/lib64/Revo-7.3/R-3.1.1/lib64/R/library/rJava’
The downloaded source packages are in
‘/tmp/RtmpY0FHgS/downloaded_packages’
Updating HTML index of packages in '.Library'
Making 'packages.html' ... done
Warning message:
In install.packages("rJava") :
  installation of package ‘rJava’ had non-zero exit status

Solution:
Here java libs are pointing incorrect location, ideally should display like:
ln -s /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71.x86_64/jre/lib/amd64/server/libjvm.so /usr/lib64/libjvm.so
run R CMD javareconf ( To refresh java config after creation of soft link)

To find:
Location: /usr/lib64/Revo-7.3/R-3.1.1/lib64/R/etc/ --check for any files updated by today
find /usr/ | grep libjvm.so
rm /usr/lib64/libjvm.so
ls -la /usr/lib64/libhdfs.so
/usr/lib64/libhdfs.so -> /usr/hdp/2.2.0.0-2041/usr/lib/libhdfs.so.0.0.0 ( Incorrect Link)

--------------------
Problem:
rhive.connect(host="00.000.00.00",port=10000,defaultFS="hdfs://00.000.00.00:8020")
[Fatal Error] hadoop-env.sh.xml:2:1: Content is not allowed in prolog.
2015-06-20 22:09:16,608 FATAL [main] conf.Configuration (Configuration.java:loadResource(2518)) - error parsing conf file:/etc/hadoop/conf/hadoop-env.sh.xml
org.xml.sax.SAXParseException; systemId: file:/etc/hadoop/conf/hadoop-env.sh.xml; lineNumber: 2; columnNumber: 1; Content is not allowed in prolog.
org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:299)
 file:/etc/hadoop/conf/hadoop-env.sh.xml; lineNumber: 2; columnNumber: 1; Content is not allowed in prolog.

Solution: remove unwanted hadoop-env.sh.xml from /etc/hadoop/conf/
-----------
> install.packages("RCurl", repos="http://cran.r-project.org")
Installing package into ‘/usr/lib64/R/library’
(as ‘lib’ is unspecified)
trying URL 'http://cran.r-project.org/src/contrib/RCurl_1.95-4.7.tar.gz'
Content type 'application/x-gzip' length 916897 bytes (895 KB)
* installing *source* package ‘RCurl’ ...
** package ‘RCurl’ successfully unpacked and MD5 sums checked
checking for curl-config... no
Cannot find curl-config
ERROR: configuration failed for package ‘RCurl’
* removing ‘/usr/lib64/R/library/RCurl’
The downloaded source packages are in
        ‘/tmp/Rtmp0CKyl3/downloaded_packages’
Updating HTML index of packages in '.Library'
Making 'packages.html' ... done
Warning message:
In install.packages("RCurl", repos = "http://cran.r-project.org") :
  installation of package ‘RCurl’ had non-zero exit status
>


Saturday, March 14, 2015

R Programming :11 Pi Charts,

Simple Pi Charts in R:
> slices <- c(10,15,20,25,8)
> lbls <- c("US","UK","AUS","GER","FRA")
> pie(slices,labels=lbls,main="Pi")

Thursday, March 12, 2015

R Programming :10 Binomial Distribution in R


R Programming 9: Correlation and Covariance in R

Correlation:
Correlation is a statistical technique that can show whether and how strongly pairs of variables are related

Ex:
Height and Weight are related.taller people tend to be heavier than shorter people.

Types of Correlation:
1) Pearson
2) Product-Moment correlation.

The main result of a correlation is called the correlation coefficient.it ranges from -1.0 to +1.0. The closed r is to +1 or -1, the more closely the two variables are related.

If r is close to 0, it means there is no relationship between the variables. If r is positive, it means that as one variable gets larger the other gets larger. If r is negative it means that as one gets larger, the other gets smaller (often called an "inverse" correlation).

R Command:
> cor(speed,dist)
[1] 0.8068949
> cor(speed,dist,method="spearman")
[1] 0.8303568
> cor(speed,dist,method="kendall")
[1] 0.6689901
> cor.test(speed,dist,method="pearson")
 Pearson's product-moment correlation
data:  speed and dist
t = 9.464, df = 48, p-value = 1.49e-12
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
 0.6816422 0.8862036
sample estimates:
      cor 
0.8068949 
 cor.test(speed,dist,method="spearman",exact=F)
 Spearman's rank correlation rho
data:  speed and dist
S = 3532.819, p-value = 8.825e-14
alternative hypothesis: true rho is not equal to 0
sample estimates:
      rho 
0.8303568 
> cor.test(speed,dist,method="pearson",alt="greater",conf.level=0.99)
 Pearson's product-moment correlation
data:  speed and dist
t = 9.464, df = 48, p-value = 7.45e-13
alternative hypothesis: true correlation is greater than 0
99 percent confidence interval:
 0.6519786 1.0000000
sample estimates:
      cor 
0.8068949 

Covariance:
Covariance indicates how two variables are related. A positive covariance means the variables are positively related, while a negative covariance means the variables are inversely related.






Wednesday, March 11, 2015

R Programming 8 : Mean, Standard Deviation,


Mean:The 'mean' is the 'average you're used to, where you add up all the numbers and then
divide by the number of numbers.

Find the mean, median, mode, and range for the following values:
10, 11, 12, 13, 14, 15, 16, 17, 18

The mean is the usual average, so:

(10+11+12+13+14+15+16+17+18) ÷ 9 = 14

R Command:
> mean(cars$speed)
[1] 15.4

Median: The median is the middle value.
First Sort the Values in increasing Order then take middle value
Ex:
1,5,3,7,2,6,10,8
Sorting: 1,2,3,6,7,8,10
Median Value = 6

R Command:
> median(cars$speed)
[1] 15

Range:
The largest value is 13 and Smallest value is 6, so the range is 13-6 =7
> range(cars$speed)
[1]  4 25

Mode:
The mode is the number repeated most often. This list has two values that are
repeated three times.
library(doBy)
library(dplyr)

Tuesday, March 10, 2015

R Programming 7: Normal Distribution

Normal Distribution:
Density, distribution function, quantile function and random generation for the normal distribution with mean equal to mean and standard deviation equal to sd

R Programming 6: R on Hadoop Hive




Connection:
-> library(RHive)
Loading required package: rJava
Loading required package: Rserve
-> rhive.init(hiveHome="/usr/hdp/current/hive-client/",hadoopHome="/usr/hdp/current/hadoop-client")
-> rhive.connect(host="HS2",port=10000,defaultFS="hdfs://HiveCLI/R server:8020")

Extensions in R:
rhive.connect
fhive.query
rhive.assign
rhive.export
rhive.napply
rhive.sapply
rhive.aggregate
rhive.list.tables
rhive.load.table
rhive.desc.table
Ex:
rhive.desc.table("diva.tablename")

Setting hive.execution.engine as tez in R:
rhive.set('hive.execution.engine','tez')

input <- rhive.query("select * from db.tableanme limit 10")


Issues:

> hive.query("show tables")
Error: could not find function "hive.query"
> library(RHive)
> rhive.init(hiveHome="/usr/hdp/current/hive-client/",hadoopHome="/usr/hdp/current/hadoop-client")
> rhive.connect(host="HiveServer2",port=10000,defaultFS="hdfs://hiveClient:8020"
+ hive.query("show tables")
Error: unexpected symbol in:
"rhive.connect(host="HiveServer2",port=10000,defaultFS="hdfs://iveClient:8020"
hive.query"

2015-03-11 20:55:19,572 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-03-11 20:55:20,281 WARN  [main] shortcircuit.DomainSocketFactory (DomainSocketFactory.java:<init>(116)) - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
Warning:
+----------------------------------------------------------+
+ / hiveServer2 argument has not been provided correctly.  +
+ / RHive will use a default value: hiveServer2=TRUE.      +
+----------------------------------------------------------+

2015-03-11 20:55:20,615 INFO  [Thread-4] jdbc.Utils (Utils.java:parseURL(285)) - Supplied authorities: HS2:10000


Monday, March 9, 2015

R Programming 5: Histogram ,Bar and Pie Charts

A histogram consists of parallel vertical bars that graphically shows the frequency distribution of a quantitative variable. The area of each bar is equal to the frequency of items found in each class.

Example
In the data set Marks, the histogram of the eruptions variable is a collection of parallel vertical bars showing the number of eruptions classified according to their durations.

Ex Syntax:
hist(input$grades,breaks=seq(from=0,to=11,by=1), main="Histogram of Grades",col.main="mediumblue",xlab="Grades",las=1,col=c("red","yellow","green","violet","orange","blue","pink","cyan","brown","bisque3"))

Friday, March 6, 2015

SMTP Connection in Hadoop Cluster



vi mailtestdiva.sh
./mailtestdiva.sh
 mailq
 tailf /var/log/maillog
vi /etc/postfix/main.cf
 service postfix restart
./mailtestdiva.sh
tailf /var/log/maillog
 ./mailtestdiva.sh
tailf /var/log/maillog

/etc/postfix/main.cf ( this is other client machine not on Oozie Server Machine)
relayhost = [OozieServerHost]

service postfix restart


on Oozie Server machine
/etc/postfix/main.cf

relayhost = [smtp.fmi.com]

# Enable IPv4, and IPv6 if supported
inet_protocols = all



1029  mail --help
 1030  man mail
 1031  cd test/
 1032  ls -lrt
 1033  cd
 1034  cd divakar/
 1035  ls -lrt
 1036  cd
 1037  cat mailtestdiva.sh
 1038  cat /etc/resolv.conf
 1039  ./mailtestdiva.sh
 1040  cat /var/spool/mail/root
 1041  ls
 1042  cd /var/log/
 1043  ls
 1044  less messages
 1045  tailf /var/log/secure
 1046  tail /var/log/maillog
 1047  host smtp.fmi.com
 1048  vim /etc/mail.rc
 1049  vi /etc/mail.rc
 1050  vi /etc/postfix/main.cf
 1051  service postfix status
 1052  service postfix restart
 1053  tailf /var/log/maillog
 1054  ls
 1055  cd /etc/postfix/
 1056  vim main.cf
 1057  vi main.cf
 1058  service postfix restart
 1059  tailf /var/log/maillog
 1060  ls -lrt
 1061  vi main.cf
 1062  history

Connect teradata from R