Talend Sqoop is split my free form queries into different varibale

96 views Asked by At

I have this bash script.

JDBC_CON=$1
USERNAME=$2
PASSWORD=$3
HIVE_DB=$4
TABLE_NAME=$5
QUERY=$6
maxlength=500
COLUMN=$7
TARGET_DIR=$8
MAPPER=$9

echo $QUERY

sqoop import -Dorg.apache.sqoop.splitter.allow_text_splitter=true \
--connect $JDBC_CON --username $USERNAME --password $PASSWORD \
--hive-import --hive-database $HIVE_DB --hive-table $TABLE_NAME \
--fetch-size 1000 --delete-target-dir --compression-codec org.apache.hadoop.io.compress.SnappyCodec \
--as-textfile --query "$QUERY" \
--split-by $COLUMN --target-dir $TARGET_DIR -m $MAPPER;

When i run this via Talend Tsystem, the varible Query gets splitted into different variables.

Sharing log till the breakage.

INFO ] 08:21:52 xxx_xxx.import_data_0_6.Import_Data- TalendJob: 'Import_Data' - Start. [statistics] connecting to socket on port 10739 [statistics] connected [INFO ] 08:21:52 aycap_dev.import_data_0_6.Import_Data- tFileInputDelimited_1 - Retrieving records from the datasource. [INFO ] 08:21:52 aycap_dev.import_data_0_6.Import_Data- tSystem_1 - Setting the parameters. [INFO ] 08:21:52 aycap_dev.import_data_0_6.Import_Data- tSystem_1 - Root directory: '/home/talenduser/dm/'. [INFO ] 08:21:52 aycap_dev.import_data_0_6.Import_Data- tSystem_1 - Executing the command. [INFO ] 08:21:52 aycap_dev.import_data_0_6.Import_Data- tSystem_1 - Command to execute: 'sh -x import_data.sh xxxxxx talend xxxxxx default tablename SELECT /* FROM tablename WHERE $CONDITIONS column1 /user/hive/tmp '. [INFO ] 08:21:52 aycap_dev.import_data_0_6.Import_Data- tSystem_1 - The command has been executed successfully. SELECT Warning: /opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation.

  • JDBC_CON=jdbc:oracle:thin:@//xxxxxxx/orcl
  • USERNAME=talend
  • PASSWORD=xxxxx
  • HIVE_DB=default
  • TABLE_NAME=tablename
  • QUERY=SELECT
  • COLUMN='*'
  • TARGET_DIR=FROM
  • MAPPER=TALEND.tablename
  • echo SELECT
  • sqoop import -Dorg.apache.sqoop.splitter.allow_text_splitter=true --connect xxxxxx --username talend --password xxxxx --hive-import --hive-database default--hive-table xxxxx --fetch-size 1000 --delete-target-dir --compression-codec org.apache.hadoop.io.compress.SnappyCodec --as-textfile --query SELECT --split-by /bin /boot /data /dev /etc /home /keytab /lib /lib64 /media /mnt /opt /proc /root /run /sbin /srv /sys /tmp /usr /var --target-dir FROM -m tablename

Can someone help me on this.

0

There are 0 answers