Thursday, September 25, 2014

Apache Sqoop -Part 3: Sqoop data Import & export and Interview Questions


Using Teradata Drivers:

sqoop import \
--connect jdbc:teradata://****/DATABASE=****,TYPE=FASTEXPORT,CHARSET=UTF8 \
--driver "com.teradata.jdbc.TeraDriver" \
--username ****** \
--password **** \
--query "SELECT xyz,abc,jkh, where date >= '2015-01-16' and date < '2015-02-01' AND \$CONDITIONS" \
--hive-table diva.entry \
--create-hive-table \
--hive-import \
--target-dir /diva/entry54214 \
--split-by id \
--fields-terminated-by '|' \
--m 10;

Ex:
To Import as Text File using Teradata drivers
sqoop import -Dhdp.version=2.5.3.0-37 \ --connect jdbc:teradata://xxxxxxx.org/DATABASE=xxxx \ --connection-manager org.apache.sqoop.teradata.TeradataConnManager \ --username xxxxxxxx \ --password xxxxxxx \ --table STATUS \ --target-dir /tmp/tt/status \ --as-textfile


Oracle :

sqoop export --connect jdbc:-Oracle:thin:@IP ADDRESS:1555:xwdasnvo --password ****** --username ****** --table *******.SQOOP_TEST --columns SSID --export-dir /user/hive/warehouse/wifi

Mysql:
sqoop import --connect jdbc:mysql://master/test --table SqoopTest;
sqoop list-databases --connect jdbc:mysql://master


Issues:
ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: No columns to generate for ClassWriter
[Error 9804] [SQLState HY000] Response Row size or Constant Row size overflow
Error executing statement: com.teradata.jdbc.jdbc_4.util.JDBCException: [Teradata Database] [TeraJDBC 14.10.00.26] [Error 9804] [SQLState HY000] Response Row size or Constant Row size overflow
ERROR manager.SqlManager: Error executing statement: com.teradata.jdbc.jdbc_4.util.JDBCException: [Teradata Database] [TeraJDBC 14.10.00.26] [Error 9804]

Solution: set session charset 'ascii';
Ex:
sqoop import \
--connect jdbc:teradata://DBNAME/DATABASE=schemaname,CHARSET=ascii \
--driver "com.teradata.jdbc.TeraDriver" \
--username XXXXXXXX \
--password XXXXXXXX \
--query "select * from XXXXXXXX where \$CONDITIONS" \
--split-by <any interger column> \
--target-dir /path/to/hdfs/ \
--fields-terminated-by '\t' \
--m 4
-------------------------------------------
Issue:
ERROR manager.SqlManager: Error executing statement: com.microsoft.sqlserver.jdbc.SQLServerException: Invalid object name.

Solution: Hadoop user is not able to find the Target Schema or it's always pointing to default DB.


No comments:

Post a Comment