sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dipesh Kumar Singh <dipesh.t...@gmail.com>
Subject Re: Sqoop export to Teradata
Date Thu, 10 Oct 2013 03:41:36 GMT
Thanks a ton Jarek.

I am able to rectify that and run the command but it kills all the
efficiency sqoop provides for bulk load.

sqoop export -Dsqoop.export.records.per.statement=1 \
--verbose \
--driver com.teradata.jdbc.TeraDriver \
--connect jdbc:teradata://
hostname.com/DATABASE=DW1_DAWS_WORK,TMODE=ANSI,LOGMECH=LDAP \
--export-dir /user/dipeshks/T_DLY_sqoop \
--table T_DLY_d   \
--input-fields-terminated-by '|' \
--username myuser \
--num-mappers 1 \
--batch
-P

I wanted to know few things :

1> Is there some way, i can plug some custom code with sqoop and use
Teradata FastLoad capabilities. It would be help if anyone can direct be to
some references to start with.


2> I would like to have some explaination behind increase in number of
mappers causing this deadlock error:

java.io.IOException: com.teradata.jdbc.jdbc_4.util.JDBCException: [Teradata
Database] [TeraJDBC 13.00.00.07] [Error 2631] [SQLState 40001] Transaction
ABORTed due to deadlock.
 at
org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:220)
at
org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:46)
 at
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:531)
at
org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
 at
org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:78)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:36)
 at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at
org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:182)
 at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
 at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
at java.


Thanks & Regards,
Dipesh


On Mon, Sep 30, 2013 at 10:39 PM, Jarek Jarcec Cecho <jarcec@apache.org>wrote:

> Hi Dipesh,
> Sqoop by default will generate insert statements that have multiple rows
> in format:
>
>   INSERT INTO table VALUES(), (), (), ...
>
> This particular SQL extension is very common in database world, but it's
> unfortunately not supported by Teradata. As a result you have to turn it
> off by setting property sqoop.export.records.per.statement to 1, for
> example:
>
>   sqoop export -Dsqoop.export.records.per.statement=1 --connect ...
>
> Jarcec
>
> On Sat, Sep 28, 2013 at 10:13:00PM +0530, Dipesh Kumar Singh wrote:
> > Hello Users,
> >
> > I am getting the following error while exporting the data from HDFS to
> > Teradata.
> >
> > *Sqoop Command used : *
> > *
> > *
> > sqoop export --verbose \
> > --driver com.teradata.jdbc.TeraDriver \
> > --connect jdbc:teradata://
> > hostname.domain.com/DATABASE=DW1_DAWS_WORK,TMODE=ANSI,LOGMECH=LDAP \
> > --export-dir /user/myusername/TDETAILS_ADDR_CLT_DLY_sqoop \
> > --table TDETAILS_ADDR_CLT_DLY_d   \
> > --input-fields-terminated-by '|' \
> > --input-escaped-by '\\' \
> > //--lines-terminated-by '\n' \
> > --username myusername \
> > --num-mappers 8 \
> > -P
> >
> > Though, i am successfully able to do sqoop import. The output of which is
> > /user/myusername/TDETAILS_ADDR_CLT_DLY_sqoop
> >
> > *ERROR snip / Stack trace on Sqoop export:*
> > *
> > *
> > 13/09/27 09:20:37 INFO mapred.JobClient: Running job:
> job_201309191609_5631
> > 13/09/27 09:20:38 INFO mapred.JobClient:  map 0% reduce 0%
> > 13/09/27 09:20:45 INFO mapred.JobClient: Task Id :
> > attempt_201309191609_5631_m_000000_0, Status : FAILED
> > java.io.IOException: com.teradata.jdbc.jdbc_4.util.JDBCException:
> [Teradata
> > Database] [TeraJDBC 13.00.00.07] [Error 3706] [SQLState 42000] Syntax
> > error: expected something between ')' and ','.
> >         at
> >
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:220)
> >         at
> >
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:46)
> >         at
> >
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:531)
> >         at
> >
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> >         at
> > org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:78)
> >         at
> > org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:36)
> >         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> >         at
> >
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:182)
> >         at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
> >         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
> >         at org.apache.hadoop.mapred.Child$4.run(Child.ja
> > 13/09/27 09:20:46 INFO mapred.JobClient: Task Id :
> > attempt_201309191609_5631_m_000004_0, Status : FAILED
> > java.io.IOException: com.teradata.jdbc.jdbc_4.util.JDBCException:
> [Teradata
> > Database] [TeraJDBC 13.00.00.07] [Error 3706] [SQLState 42000] Syntax
> > error: expected something between ')' and ','.
> >         at
> >
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:220)
> >         at
> >
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:46)
> >         at
> >
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:531)
> >         at
> >
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> >         at
> > org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:78)
> >         at
> > org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:36)
> >         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> >         at
> >
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:182)
> >         at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
> >         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
> >         at org.apache.hadoop.mapred.Child$4.run(Child.ja
> > 13/09/27 09:20:46 INFO mapred.JobClient: Task Id :
> > attempt_201309191609_5631_m_000005_0, Status : FAILED
> > java.io.IOException: com.teradata.jdbc.jdbc_4.util.JDBCException:
> [Teradata
> > Database] [TeraJDBC 13.00.00.07] [Error 3706] [SQLState 42000] Syntax
> > error: expected something between ')' and ','.
> >         at
> >
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:220)
> >         at
> >
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:46)
> >         at
> >
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:531)
> >         at
> >
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> >         at
> > org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:78)
> >         at
> > org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:36)
> >         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> >         at
> >
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:182)
> >         at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
> >         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
> >         at org.apache.hadoop.mapred.Child$4.run(Child.ja
> >
> > *
> > *
> >
> > Seems like Sqoop is not able to generate correct insert query for TD.
> >
> > Has anyone faced this issue? Any ideas on how to overcome it ?
> >
> >
> > Thanks & Regards,
> > Dipesh
> > --
> > Dipesh Kr. Singh
>



-- 
Dipesh Kr. Singh

Mime
View raw message