sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "arvind@cloudera.com" <arv...@cloudera.com>
Subject Re: [sqoop-user] Sqoop-with Terradata
Date Fri, 09 Sep 2011 18:07:11 GMT
Hi Srinivas,

I am sorry that you are running into these issues. I think you may be
hitting [SQOOP-314]. We have recently fixed it on the trunk, so can
you please try out a build from the trunk to see if it resolves your
problem. You should run the export with the --batch option.

Also, to subscribe to sqoop-user@incubator.apache.org, you would first
need to send a mail to incubator-sqoop-user-subscribe@apache.org and
follow the instructions that come back.

[SQOOP-314] https://issues.apache.org/jira/browse/SQOOP-314

Thanks,
Arvind

On Thu, Sep 8, 2011 at 8:30 PM, SRINIVAS SURASANI <vasajb@gmail.com> wrote:
> Arvind,
> I have tried sending an reply to sqoop-user@incubator.apache.org but its
> giving me failure notification.
>
> Sqoop export is hanging up after 100% of map is done. Even  I have tried
> with "sqoop.export.records.per.statement"=1 still after 100% of map, job
> goes into idle condition.Importing into HDFS from Terradata is working
> fine.
>
> Thanks,
> Srini
> On Wed, Aug 31, 2011 at 12:19 AM, SRINIVAS SURASANI <vasajb@gmail.com>
> wrote:
>>
>> Arvind,
>> Here we have two scenarios.
>> 1.The xml data as it is processed need to be loaded into Terradata ( this
>> is for analysis of intra-day).  So, we will be having around 3500 files/day
>> to be loaded into Terrdata. This is like semi-real kind of loading. Duration
>> between source landing(xml data into HDFS) and loading(csv files into
>> Terradata) is couple of minutes.
>> 2. As a batch. Meaning, proceesing entire day data together ( xml to csv
>> using Pig code) and sending this single directory as a batch.
>>
>> The scenarios might change.. as the project is in early stages.
>> Regards,
>> Srini
>>
>> On Tue, Aug 30, 2011 at 11:48 PM, arvind@cloudera.com
>> <arvind@cloudera.com> wrote:
>>>
>>> Thanks Srini. What kind of work load do you have?
>>>
>>> - Arvind
>>>
>>> On Tue, Aug 30, 2011 at 5:47 PM, SRINIVAS SURASANI <vasajb@gmail.com>
>>> wrote:
>>> > Arvind,
>>> > I'm working as contractor at MorganStanley.
>>> >
>>> > Srini
>>> > On Tue, Aug 30, 2011 at 5:41 PM, arvind@cloudera.com
>>> > <arvind@cloudera.com>
>>> > wrote:
>>> >>
>>> >> Hi Srini,
>>> >>
>>> >> I am glad I was able to help. Do you mind sharing with me some more
>>> >> details regarding how/where you are using Sqoop? I keep a track of the
>>> >> various deployments in order to help prioritize the feature
>>> >> development so that it is aligned with what the users need rather than
>>> >> what we want to do first.
>>> >>
>>> >> It will be great if you can give me an idea of which company this
>>> >> project was for and what was the high-level workload like, as long as
>>> >> it is public information.
>>> >>
>>> >> Thanks,
>>> >> Arvind
>>> >>
>>> >>
>>> >> ---------- Forwarded message ----------
>>> >> From: SRINIVAS SURASANI <vasajb@gmail.com>
>>> >> Date: Mon, Aug 29, 2011 at 3:06 PM
>>> >> Subject: Re: [sqoop-user] Sqoop-with Terradata
>>> >> To: sqoop-user@cloudera.org
>>> >>
>>> >>
>>> >> Arvind,
>>> >>
>>> >> Thank-you very much. I'm able to connect to Terradata.
>>> >>
>>> >>
>>> >> Srini
>>> >>
>>> >> On Mon, Aug 29, 2011 at 2:58 PM, arvind@cloudera.com
>>> >> <arvind@cloudera.com> wrote:
>>> >> >
>>> >> > This may be related to not having the GSS configuration classes
in
>>> >> > your classpath. I would suggest you add tdgssconfig.jar to the
>>> >> > classpath and try again.
>>> >> >
>>> >> > Thanks,
>>> >> > Arvind
>>> >> >
>>> >> > On Mon, Aug 29, 2011 at 11:19 AM, SRINIVAS SURASANI
>>> >> > <vasajb@gmail.com>
>>> >> > wrote:
>>> >> > > Arvind,
>>> >> > >
>>> >> > > I have subscribed at sqoop-user@incubator.apache.org and posted
>>> >> > > Question .
>>> >> > > Sorry for the inconvinence from my end, since I'm close to
>>> >> > > deadline
>>> >> > > I'm
>>> >> > > taking your valuable time.
>>> >> > >
>>> >> > > sqoop list-tables --driver com.teradata.jdbc.TeraDriver --connect
>>> >> > > jdbc:teradata://PKTD/E1_CMS_WORK --username srini -P
>>> >> > > Iam getting the following error:
>>> >> > > 11/08/29 13:08:03 INFO manager.SqlManager: Using default fetchSize
>>> >> > > of
>>> >> > > 1000
>>> >> > > GSSException: Failure unspecified at GSS-API level (Mechanism
>>> >> > > level:
>>> >> > > UserFile parameter null)
>>> >> > >         at
>>> >> > > com.teradata.tdgss.jtdgss.TdgssParseXml.<init>(DashoA1*..)
>>> >> > >         at
>>> >> > > com.teradata.tdgss.jtdgss.TdgssConfigApi.<init>(DashoA1*..)
>>> >> > >         at
>>> >> > > com.teradata.tdgss.jtdgss.TdgssManager.<init>(DashoA1*..)
>>> >> > >         at
>>> >> > > com.teradata.tdgss.jtdgss.TdgssManager.getInstance(DashoA1*..)
>>> >> > >         at
>>> >> > >
>>> >> > >
>>> >> > > com.teradata.jdbc.jdbc.GenericTeraEncrypt.getGSSM(GenericTeraEncrypt.java:612)
>>> >> > >         at
>>> >> > >
>>> >> > >
>>> >> > > com.teradata.jdbc.jdbc.GenericTeraEncrypt.getConfig(GenericTeraEncrypt.java:630)
>>> >> > >         at
>>> >> > >
>>> >> > >
>>> >> > > com.teradata.jdbc.jdbc.GenericTeraEncrypt.getUserNameForOid(GenericTeraEncrypt.java:723)
>>> >> > >         at
>>> >> > > com.teradata.jdbc.AuthMechanism.<init>(AuthMechanism.java:50)
>>> >> > >         at
>>> >> > >
>>> >> > >
>>> >> > > com.teradata.jdbc.jdbc.GenericInitDBConfigState.action(GenericInitDBConfigState.java:105)
>>> >> > >         at
>>> >> > >
>>> >> > >
>>> >> > > com.teradata.jdbc.jdbc.GenericLogonController.run(GenericLogonController.java:49)
>>> >> > >         at
>>> >> > > com.teradata.jdbc.jdbc_4.TDSession.<init>(TDSession.java:199)
>>> >> > >         at
>>> >> > >
>>> >> > >
>>> >> > > com.teradata.jdbc.jdbc_3.ifjdbc_4.TeraLocalConnection.<init>(TeraLocalConnection.java:95)
>>> >> > >         at
>>> >> > >
>>> >> > >
>>> >> > > com.teradata.jdbc.jdbc.ConnectionFactory.createConnection(ConnectionFactory.java:54)
>>> >> > >         at
>>> >> > > com.teradata.jdbc.TeraDriver.doConnect(TeraDriver.java:217)
>>> >> > >         at
>>> >> > > com.teradata.jdbc.TeraDriver.connect(TeraDriver.java:150)
>>> >> > >         at
>>> >> > > java.sql.DriverManager.getConnection(DriverManager.java:582)
>>> >> > >         at
>>> >> > > java.sql.DriverManager.getConnection(DriverManager.java:185)
>>> >> > >         at
>>> >> > >
>>> >> > >
>>> >> > > com.cloudera.sqoop.manager.SqlManager.makeConnection(SqlManager.java:643)
>>> >> > >         at
>>> >> > >
>>> >> > >
>>> >> > > com.cloudera.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:51)
>>> >> > >         at
>>> >> > >
>>> >> > > com.cloudera.sqoop.manager.SqlManager.listTables(SqlManager.java:270)
>>> >> > >         at
>>> >> > > com.cloudera.sqoop.tool.ListTablesTool.run(ListTablesTool.java:49)
>>> >> > >         at com.cloudera.sqoop.Sqoop.run(Sqoop.java:144)
>>> >> > >         at
>>> >> > > org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>> >> > >         at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:180)
>>> >> > >         at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:219)
>>> >> > >         at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:228)
>>> >> > >         at com.cloudera.sqoop.Sqoop.main(Sqoop.java:237)
>>> >> > > 11/08/29 13:08:04 ERROR sqoop.Sqoop: Got exception running
Sqoop:
>>> >> > > java.lang.NullPointerException
>>> >> > > java.lang.NullPointerException
>>> >> > >         at
>>> >> > > com.teradata.tdgss.jtdgss.TdgssConfigApi.GetMechanisms(DashoA1*..)
>>> >> > >         at
>>> >> > > com.teradata.tdgss.jtdgss.TdgssManager.<init>(DashoA1*..)
>>> >> > >         at
>>> >> > > com.teradata.tdgss.jtdgss.TdgssManager.getInstance(DashoA1*..)
>>> >> > >         at
>>> >> > >
>>> >> > >
>>> >> > > com.teradata.jdbc.jdbc.GenericTeraEncrypt.getGSSM(GenericTeraEncrypt.java:612)
>>> >> > >         at
>>> >> > >
>>> >> > >
>>> >> > > com.teradata.jdbc.jdbc.GenericTeraEncrypt.getConfig(GenericTeraEncrypt.java:630)
>>> >> > >         at
>>> >> > >
>>> >> > >
>>> >> > > com.teradata.jdbc.jdbc.GenericTeraEncrypt.getUserNameForOid(GenericTeraEncrypt.java:723)
>>> >> > >         at
>>> >> > > com.teradata.jdbc.AuthMechanism.<init>(AuthMechanism.java:50)
>>> >> > >         at
>>> >> > >
>>> >> > >
>>> >> > > com.teradata.jdbc.jdbc.GenericInitDBConfigState.action(GenericInitDBConfigState.java:105)
>>> >> > >         at
>>> >> > >
>>> >> > >
>>> >> > > com.teradata.jdbc.jdbc.GenericLogonController.run(GenericLogonController.java:49)
>>> >> > >         at
>>> >> > > com.teradata.jdbc.jdbc_4.TDSession.<init>(TDSession.java:199)
>>> >> > >         at
>>> >> > >
>>> >> > >
>>> >> > > com.teradata.jdbc.jdbc_3.ifjdbc_4.TeraLocalConnection.<init>(TeraLocalConnection.java:95)
>>> >> > >         at
>>> >> > >
>>> >> > >
>>> >> > > com.teradata.jdbc.jdbc.ConnectionFactory.createConnection(ConnectionFactory.java:54)
>>> >> > >         at
>>> >> > > com.teradata.jdbc.TeraDriver.doConnect(TeraDriver.java:217)
>>> >> > >         at
>>> >> > > com.teradata.jdbc.TeraDriver.connect(TeraDriver.java:150)
>>> >> > >         at
>>> >> > > java.sql.DriverManager.getConnection(DriverManager.java:582)
>>> >> > >         at
>>> >> > > java.sql.DriverManager.getConnection(DriverManager.java:185)
>>> >> > >         at
>>> >> > >
>>> >> > >
>>> >> > > com.cloudera.sqoop.manager.SqlManager.makeConnection(SqlManager.java:643)
>>> >> > >         at
>>> >> > >
>>> >> > >
>>> >> > > com.cloudera.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:51)
>>> >> > >         at
>>> >> > >
>>> >> > > com.cloudera.sqoop.manager.SqlManager.listTables(SqlManager.java:270)
>>> >> > >         at
>>> >> > > com.cloudera.sqoop.tool.ListTablesTool.run(ListTablesTool.java:49)
>>> >> > >         at com.cloudera.sqoop.Sqoop.run(Sqoop.java:144)
>>> >> > >         at
>>> >> > > org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>> >> > >         at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:180)
>>> >> > >         at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:219)
>>> >> > >         at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:228)
>>> >> > >         at com.cloudera.sqoop.Sqoop.main(Sqoop.java:237)
>>> >> > > Thanks,
>>> >> > > Srini
>>> >> > >
>>> >> > >
>>> >> > >
>>> >> > > On Mon, Aug 29, 2011 at 12:58 PM, arvind@cloudera.com
>>> >> > > <arvind@cloudera.com>
>>> >> > > wrote:
>>> >> > >>
>>> >> > >> [Please subscribe and respond to sqoop-user@incubator.apache.org]
>>> >> > >>
>>> >> > >> Please use HADOOP_CLASSPATH instead of CLASSPATH. Also,
in order
>>> >> > >> to
>>> >> > >> use the generic JDBC connector, you will have to specify
the
>>> >> > >> driver
>>> >> > >> class explicitly via the command line option --driver
>>> >> > >> com.teradata.jdbc.TeraDriver.
>>> >> > >>
>>> >> > >> Thanks,
>>> >> > >> Arvind
>>> >> > >>
>>> >> > >> On Mon, Aug 29, 2011 at 9:53 AM, SRINIVAS SURASANI
>>> >> > >> <vasajb@gmail.com>
>>> >> > >> wrote:
>>> >> > >> > Arvind,
>>> >> > >> > I have set the classpath to teradata4.jar [ not placed
the
>>> >> > >> > teradata4.jar
>>> >> > >> > in
>>> >> > >> > sqoop lib, as I dont have permissions].
>>> >> > >> > I'm getting the following error
>>> >> > >> >
>>> >> > >> > sqoop list-tables --connect jdbc:teradata://PKTD/E1_CMS_WORK
>>> >> > >> > --username
>>> >> > >> > srini -P
>>> >> > >> > ERROR: tool.BaseSqoopTool: Got error creating database
manager:
>>> >> > >> > java.io.IOexception: No manager for connect string:
>>> >> > >> > jdbc:teradata:///PKTD/E1_CMS_WORK
>>> >> > >> >    at
>>> >> > >> > com.cloudera.sqoop.ConnFactory.getManager(ConnFactory.java:119)
>>> >> > >> >    at
>>> >> > >> >
>>> >> > >> > com.cloudera.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:186)
>>> >> > >> >    ...
>>> >> > >> >    ...
>>> >> > >> > Thanks,
>>> >> > >> > Srini
>>> >> > >> >
>>> >> > >> >
>>> >> > >> >
>>> >> > >> > On Mon, Aug 29, 2011 at 8:52 AM, SRINIVAS SURASANI
>>> >> > >> > <vasajb@gmail.com>
>>> >> > >> > wrote:
>>> >> > >> >>
>>> >> > >> >> Thanks-a-lot Arvind.
>>> >> > >> >>
>>> >> > >> >> On Mon, Aug 29, 2011 at 8:45 AM, arvind@cloudera.com
>>> >> > >> >> <arvind@cloudera.com>
>>> >> > >> >> wrote:
>>> >> > >> >>>
>>> >> > >> >>> [Moving the thread to sqoop-user@incubator.apache.org]
>>> >> > >> >>>
>>> >> > >> >>> Hi Srini,
>>> >> > >> >>>
>>> >> > >> >>> You should be able to use the generic JDBC
connector to
>>> >> > >> >>> import/export
>>> >> > >> >>> from Teradata. There is also a specialized
connector that is
>>> >> > >> >>> available
>>> >> > >> >>> for use with Teradata if you are interested.
This connector
>>> >> > >> >>> is
>>> >> > >> >>> not a
>>> >> > >> >>> part of Sqoop and can be obtained from Cloudera
by going to:
>>> >> > >> >>>
>>> >> > >> >>> http://www.cloudera.com/partners/connectors/
>>> >> > >> >>>
>>> >> > >> >>> Thanks,
>>> >> > >> >>> Arvind
>>> >> > >> >>>
>>> >> > >> >>> On Mon, Aug 29, 2011 at 8:17 AM, SRINIVAS
SURASANI
>>> >> > >> >>> <vasajb@gmail.com>
>>> >> > >> >>> wrote:
>>> >> > >> >>> > I have csv file in hadoop and looking
to load into
>>> >> > >> >>> > Teradata. I
>>> >> > >> >>> > was
>>> >> > >> >>> > wondering does the sqoop works with
Terradata.(with JDBC
>>> >> > >> >>> > jar
>>> >> > >> >>> > placing
>>> >> > >> >>> > in sqoop lib dir).
>>> >> > >> >>> >
>>> >> > >> >>> > Regards
>>> >> > >> >>> > Srini
>>> >> > >> >>> >
>>> >> > >> >>> > --
>>> >> > >> >>> > NOTE: The mailing list sqoop-user@cloudera.org
is
>>> >> > >> >>> > deprecated in
>>> >> > >> >>> > favor
>>> >> > >> >>> > of Apache Sqoop mailing list
>>> >> > >> >>> > sqoop-user@incubator.apache.org.
>>> >> > >> >>> > Please
>>> >> > >> >>> > subscribe to it by sending an email
to
>>> >> > >> >>> > incubator-sqoop-user-subscribe@apache.org.
>>> >> > >> >>> >
>>> >> > >> >>>
>>> >> > >> >>> --
>>> >> > >> >>> NOTE: The mailing list sqoop-user@cloudera.org
is deprecated
>>> >> > >> >>> in
>>> >> > >> >>> favor
>>> >> > >> >>> of
>>> >> > >> >>> Apache Sqoop mailing list sqoop-user@incubator.apache.org.
>>> >> > >> >>> Please
>>> >> > >> >>> subscribe
>>> >> > >> >>> to it by sending an email to
>>> >> > >> >>> incubator-sqoop-user-subscribe@apache.org.
>>> >> > >> >>
>>> >> > >> >
>>> >> > >> > --
>>> >> > >> > NOTE: The mailing list sqoop-user@cloudera.org is
deprecated in
>>> >> > >> > favor of
>>> >> > >> > Apache Sqoop mailing list sqoop-user@incubator.apache.org.
>>> >> > >> > Please
>>> >> > >> > subscribe
>>> >> > >> > to it by sending an email to
>>> >> > >> > incubator-sqoop-user-subscribe@apache.org.
>>> >> > >> >
>>> >> > >>
>>> >> > >> --
>>> >> > >> NOTE: The mailing list sqoop-user@cloudera.org is deprecated
in
>>> >> > >> favor
>>> >> > >> of
>>> >> > >> Apache Sqoop mailing list sqoop-user@incubator.apache.org.
Please
>>> >> > >> subscribe
>>> >> > >> to it by sending an email to
>>> >> > >> incubator-sqoop-user-subscribe@apache.org.
>>> >> > >
>>> >> > > --
>>> >> > > NOTE: The mailing list sqoop-user@cloudera.org is deprecated
in
>>> >> > > favor
>>> >> > > of
>>> >> > > Apache Sqoop mailing list sqoop-user@incubator.apache.org.
Please
>>> >> > > subscribe
>>> >> > > to it by sending an email to
>>> >> > > incubator-sqoop-user-subscribe@apache.org.
>>> >> > >
>>> >> >
>>> >> > --
>>> >> > NOTE: The mailing list sqoop-user@cloudera.org is deprecated in
>>> >> > favor of
>>> >> > Apache Sqoop mailing list sqoop-user@incubator.apache.org. Please
>>> >> > subscribe
>>> >> > to it by sending an email to
>>> >> > incubator-sqoop-user-subscribe@apache.org.
>>> >>
>>> >> --
>>> >> NOTE: The mailing list sqoop-user@cloudera.org is deprecated in favor
>>> >> of Apache Sqoop mailing list sqoop-user@incubator.apache.org. Please
>>> >> subscribe to it by sending an email to
>>> >> incubator-sqoop-user-subscribe@apache.org.
>>> >
>>> >
>>
>
>

Mime
View raw message