sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cheolsoo Park <cheol...@cloudera.com>
Subject Re: Error while importing from Teradata to HDFS
Date Mon, 23 Apr 2012 18:57:01 GMT
Hi Dipesh,

I was looking for such an alternative because currently i don't
> have permissions to add jars in Sqoop/lib.


OK, that's unfortunate.

If none of these suggestions is feasible for your case, there is the last
thing that you can do: install Sqoop from the tarball in a directory where
you have full permissions and use it instead. As long as HADOOP_HOME is set
to the installed hadoop, everything will work.

Thanks,
Cheolsoo


On Mon, Apr 23, 2012 at 11:17 AM, Anirudh <techie.anirudh@gmail.com> wrote:

> Dipesh,
>
> It would be ideal if the jdbc driver is placed under Sqoop/lib as other
> users too would be able to leverage the same. But given your situation, use
> the -libjars option to point to the driver.
>
> Hope this helps.
>
> Thanks,
> Anirudh
>
>
> On Mon, Apr 23, 2012 at 11:01 AM, DIPESH KUMAR SINGH <
> dipesh.tech@gmail.com> wrote:
>
>> Hi Cheolsoo,
>>
>> Yeah, I know. I was looking for an alternative way by which we could tell
>> Sqoop to load jars from a location different than Sqoop/lib. I was looking
>> for such an alternative because currently i don't have
>> permissions to add jars in Sqoop/lib, though i have initiated the process.
>>
>> Thanks & Regards,
>>
>> Dipesh
>>
>>
>>
>> On Mon, Apr 23, 2012 at 11:13 PM, Cheolsoo Park <cheolsoo@cloudera.com>wrote:
>>
>>> Hi Dipesh,
>>>
>>> 12/04/23 01:02:42 ERROR sqoop.Sqoop: Got exception running Sqoop:
>>>> java.lang.RuntimeException: Could not load db driver class:
>>>> com.teradata.jdbc.TeraDriver
>>>>
>>> java.lang.RuntimeException: Could not load db driver class:
>>>> com.teradata.jdbc.TeraDrive
>>>
>>>
>>> Your Teradata jdbc jar is not found by Sqoop. The easiest way to fix it
>>> is to put *terajdbc4.jar* and *tdgssconfig.jar* in */usr/lib/sqoop/lib*.
>>>
>>> Thanks,
>>> Cheolsoo
>>>
>>> On Mon, Apr 23, 2012 at 10:32 AM, DIPESH KUMAR SINGH <
>>> dipesh.tech@gmail.com> wrote:
>>>
>>>> Hi Cheolsoo,
>>>>
>>>> As you suggested, I tried after removing "TeraDriver" line from
>>>> connectors file but that didn't worked
>>>> too. Please help..!!
>>>>
>>>> Here is the stack trace:
>>>>
>>>>>
>>>>> 12/04/23 01:02:35 DEBUG tool.BaseSqoopTool: Enabled debug logging.
>>>>>
>>>> Enter password:
>>>>>
>>>> 12/04/23 01:02:42 DEBUG util.ClassLoaderStack: Checking for existing
>>>>> class: com.cloudera.sqoop.manager.TeradataManagerFactory
>>>>>
>>>> 12/04/23 01:02:42 DEBUG util.ClassLoaderStack: Attempting to load jar
>>>>> through URL:
>>>>> jar:file:///ms/user/a/sqoop-user/Sqoop/sqoop-td-connector-1.0.5.jar!/
>>>>>
>>>> 12/04/23 01:02:42 DEBUG util.ClassLoaderStack: Previous classloader is
>>>>> java.net.URLClassLoader@27b15692
>>>>>
>>>> 12/04/23 01:02:42 DEBUG util.ClassLoaderStack: Testing class in jar:
>>>>> com.cloudera.sqoop.manager.TeradataManagerFactory
>>>>>
>>>> 12/04/23 01:02:42 DEBUG util.ClassLoaderStack: Loaded jar into current
>>>>> JVM: jar:file:///ms/user/a/sqoop-user/Sqoop/sqoop-td-connector-1.0.5.jar!/
>>>>>
>>>> 12/04/23 01:02:42 DEBUG util.ClassLoaderStack: Added classloader for
>>>>> jar /ms/user/a/sqoop-user/Sqoop/sqoop-td-connector-1.0.5.jar:
>>>>> java.net.FactoryURLClassLoader@19e3cd51
>>>>>
>>>> 12/04/23 01:02:42 DEBUG sqoop.ConnFactory: Added factory
>>>>> com.cloudera.sqoop.manager.TeradataManagerFactory in jar
>>>>> /ms/user/a/sqoop-user/Sqoop/sqoop-td-connector-1.0.5.jar specified by
>>>>> /ms/user/a/sqoop-user/Sqoop/conf/managers.d/connectors
>>>>>
>>>> 12/04/23 01:02:42 DEBUG sqoop.ConnFactory: Loaded manager factory:
>>>>> com.cloudera.sqoop.manager.TeradataManagerFactory
>>>>>
>>>> 12/04/23 01:02:42 DEBUG sqoop.ConnFactory: Loaded manager factory:
>>>>> com.cloudera.sqoop.manager.DefaultManagerFactory
>>>>>
>>>> 12/04/23 01:02:42 DEBUG sqoop.ConnFactory: Trying ManagerFactory:
>>>>> com.cloudera.sqoop.manager.TeradataManagerFactory
>>>>>
>>>> 12/04/23 01:02:42 DEBUG manager.TeradataManagerFactory: Trying with
>>>>> scheme: jdbc:teradata:
>>>>>
>>>> 12/04/23 01:02:42 INFO manager.SqlManager: Using default fetchSize of
>>>>> 1000
>>>>>
>>>> 12/04/23 01:02:42 DEBUG sqoop.ConnFactory: Instantiated ConnManager
>>>>> com.cloudera.sqoop.teradata.TeradataManager@220ca470
>>>>>
>>>> 12/04/23 01:02:42 INFO tool.CodeGenTool: Beginning code generation
>>>>>
>>>> 12/04/23 01:02:42 ERROR sqoop.Sqoop: Got exception running Sqoop:
>>>>> java.lang.RuntimeException: Could not load db driver class:
>>>>> com.teradata.jdbc.TeraDriver
>>>>>
>>>> java.lang.RuntimeException: Could not load db driver class:
>>>>> com.teradata.jdbc.TeraDriver
>>>>>
>>>>  at
>>>>> com.cloudera.sqoop.manager.SqlManager.makeConnection(SqlManager.java:634)
>>>>>
>>>> at
>>>>> com.cloudera.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:51)
>>>>>
>>>> at com.cloudera.sqoop.manager.SqlManager.execute(SqlManager.java:456)
>>>>>
>>>> at com.cloudera.sqoop.manager.SqlManager.execute(SqlManager.java:479)
>>>>>
>>>> at
>>>>> com.cloudera.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:192)
>>>>>
>>>> at
>>>>> com.cloudera.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:176)
>>>>>
>>>> at com.cloudera.sqoop.orm.ClassWriter.generate(ClassWriter.java:1017)
>>>>>
>>>> at com.cloudera.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:82)
>>>>>
>>>> at com.cloudera.sqoop.tool.ImportTool.importTable(ImportTool.java:337)
>>>>>
>>>> at com.cloudera.sqoop.tool.ImportTool.run(ImportTool.java:423)
>>>>>
>>>> at com.cloudera.sqoop.Sqoop.run(Sqoop.java:144)
>>>>>
>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>>>
>>>> at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:180)
>>>>>
>>>> at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:219)
>>>>>
>>>> at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:228)
>>>>>
>>>> at com.cloudera.sqoop.Sqoop.main(Sqoop.java:237)
>>>>>
>>>>
>>>>>
>>>> Thanks,
>>>>
>>>> Dipesh
>>>>
>>>>
>>>>
>>>> On Sun, Apr 22, 2012 at 3:01 PM, Cheolsoo Park <cheolsoo@cloudera.com>wrote:
>>>>
>>>>> Hi Dispesh,
>>>>>
>>>>> I added the line com.teradata.jdbc.TeraDriver=<path to jar file
>>>>>> terajdbc4.jar> along with
>>>>>> com.cloudera.sqoop.manager.TeradataManagerFactory=<path to
>>>>>> sqoop-td-connector-1.0.5.jar>
>>>>>> in connectors file inside managers.d directory.
>>>>>
>>>>>
>>>>> Please remove "com.teradata.jdbc.TeraDriver=<path to jar file
>>>>> terajdbc4.jar>" from your connector file.
>>>>>
>>>>> It should only contain "com.cloudera.sqoop.manager.TeradataManagerFactory=<path
>>>>> to sqoop-td-connector-1.0.5.jar>".
>>>>>
>>>>> Thanks,
>>>>> Cheolsoo
>>>>>
>>>>>
>>>>> On Sat, Apr 21, 2012 at 10:21 PM, DIPESH KUMAR SINGH <
>>>>> dipesh.tech@gmail.com> wrote:
>>>>>
>>>>>> I am trying to import the data from Teradata to hdfs referring
>>>>>> Cloudera Connector guide. I added the property
>>>>>>
>>>>>> (The Hadoop environment i am working in is a shared Hadoop
>>>>>> installation, So users are not allowed to
>>>>>> add any jars in lib directory or make changes in Sqoop-site.xml of
>>>>>> central installation.
>>>>>> However, users can use .profile i.e. user's configuration files to
>>>>>> specify CLASSPATH )
>>>>>>
>>>>>> I added the line com.teradata.jdbc.TeraDriver=<path to jar file
>>>>>> terajdbc4.jar> along with
>>>>>> com.cloudera.sqoop.manager.TeradataManagerFactory=<path to
>>>>>> sqoop-td-connector-1.0.5.jar>
>>>>>> in connectors file inside managers.d directory. In my CLASSPATH,
i
>>>>>> have added path to
>>>>>> SQOOP_CONF i.e. a Sqoop folder where i have kept my managers.d and
>>>>>> all jars.
>>>>>>
>>>>>>
>>>>>> With these I suppose i am able to connect to teradata using the
>>>>>> teradata manager factory using the teradriver as well
>>>>>> but, still i am facing this when i issue the sqoop import command
as
>>>>>> below
>>>>>>
>>>>>> sqoop import
>>>>>> -libjars <path to tdgssconfig.jar>
>>>>>> --driver com.teradata.jdbc.TeraDriver
>>>>>> --connect jdbc:teradata://xxxxxx/DATABASE=dbname
>>>>>> --username hadoop --P
>>>>>> --table SQOOP_TEST
>>>>>> --num-mappers 3 --verbose
>>>>>>
>>>>>> 12/04/13 02:29:32 DEBUG tool.BaseSqoopTool: Enabled debug logging.
>>>>>> 12/04/13 02:29:32 WARN tool.BaseSqoopTool: Setting your password
on
>>>>>> the command-line is insecure. Consider using -P instead.
>>>>>> 12/04/13 02:29:32 DEBUG util.ClassLoaderStack: Checking for existing
>>>>>> class: com.cloudera.sqoop.manager.TeradataManagerFactory
>>>>>> 12/04/13 02:29:32 DEBUG util.ClassLoaderStack: Attempting to load
jar
>>>>>> through URL: jar:file:///Sqoop/sqoop-td-connector-1.0.5.jar!/
>>>>>> 12/04/13 02:29:32 DEBUG util.ClassLoaderStack: Previous classloader
>>>>>> is java.net.URLClassLoader@6a8c436b
>>>>>> 12/04/13 02:29:32 DEBUG util.ClassLoaderStack: Testing class in jar:
>>>>>> com.cloudera.sqoop.manager.TeradataManagerFactory
>>>>>> 12/04/13 02:29:32 DEBUG util.ClassLoaderStack: Loaded jar into
>>>>>> current JVM: jar:file:///Sqoop/sqoop-td-connector-1.0.5.jar!/
>>>>>> 12/04/13 02:29:32 DEBUG util.ClassLoaderStack: Added classloader
for
>>>>>> jar /Sqoop/sqoop-td-connector-1.0.5.jar:
>>>>>> java.net.FactoryURLClassLoader@45db05b2
>>>>>> 12/04/13 02:29:32 DEBUG sqoop.ConnFactory: Added factory
>>>>>> com.cloudera.sqoop.manager.TeradataManagerFactory in jar
>>>>>> /Sqoop/sqoop-td-connector-1.0.5.jar specified by
>>>>>> /Sqoop/conf/managers.d/connectors
>>>>>> 12/04/13 02:29:32 DEBUG util.ClassLoaderStack: Checking for existing
>>>>>> class: com.teradata.jdbc.TeraDriver
>>>>>> 12/04/13 02:29:32 DEBUG util.ClassLoaderStack: Attempting to load
jar
>>>>>> through URL: jar:file:///Sqoop/lib/terajdbc4.jar!/
>>>>>> 12/04/13 02:29:32 DEBUG util.ClassLoaderStack: Previous classloader
>>>>>> is java.net.FactoryURLClassLoader@45db05b2
>>>>>> 12/04/13 02:29:32 DEBUG util.ClassLoaderStack: Testing class in jar:
>>>>>> com.teradata.jdbc.TeraDriver
>>>>>> 12/04/13 02:29:32 DEBUG util.ClassLoaderStack: Loaded jar into
>>>>>> current JVM: jar:file:///Sqoop/lib/terajdbc4.jar!/
>>>>>> 12/04/13 02:29:32 DEBUG util.ClassLoaderStack: Added classloader
for
>>>>>> jar /Sqoop/lib/terajdbc4.jar: java.net.FactoryURLClassLoader@518f5824
>>>>>> 12/04/13 02:29:32 DEBUG sqoop.ConnFactory: Added factory
>>>>>> com.teradata.jdbc.TeraDriver in jar /Sqoop/lib/terajdbc4.jar specified
by
>>>>>> /Sqoop/conf/managers.d/connectors
>>>>>> 12/04/13 02:29:32 DEBUG sqoop.ConnFactory: Loaded manager factory:
>>>>>> com.cloudera.sqoop.manager.TeradataManagerFactory
>>>>>> 12/04/13 02:29:32 ERROR tool.BaseSqoopTool: Got error creating
>>>>>> database manager: java.lang.ClassCastException:
>>>>>> com.teradata.jdbc.TeraDriver cannot be cast to
>>>>>> com.cloudera.sqoop.manager.ManagerFactory
>>>>>> at
>>>>>> com.cloudera.sqoop.ConnFactory.instantiateFactories(ConnFactory.java:90)
>>>>>> at com.cloudera.sqoop.ConnFactory.<init>(ConnFactory.java:60)
>>>>>> at com.cloudera.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:186)
>>>>>> at com.cloudera.sqoop.tool.ImportTool.init(ImportTool.java:81)
>>>>>> at com.cloudera.sqoop.tool.ImportTool.run(ImportTool.java:411)
>>>>>> at com.cloudera.sqoop.Sqoop.run(Sqoop.java:144)
>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>>>> at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:180)
>>>>>> at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:219)
>>>>>> at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:228)
>>>>>> at com.cloudera.sqoop.Sqoop.main(Sqoop.java:237)
>>>>>>
>>>>>>
>>>>>> Please help.
>>>>>>
>>>>>> Thanks & Regards,
>>>>>> --
>>>>>> Dipesh Kr. Singh
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Dipesh Kr. Singh
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> Dipesh Kr. Singh
>>
>>
>>
>>
>>
>

Mime
View raw message