sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From corbacho anthony <acorba...@nflabs.com>
Subject Re: Sqoop and OSGI
Date Fri, 21 Jun 2013 01:44:30 GMT
Hello Abe.

The think is, I didn`t install sqoop, I use it via pom on my java code.
I added on my code this:

Configuration conf = new HdfsConfiguration();
conf.set("sqoop.connection.factories",
"com.cloudera.sqoop.apache.ConnMananger");
[....]

return Sqoop.runTool(str, conf);
But still the same error.


On Thu, Jun 20, 2013 at 12:50 PM, Abraham Elmahrek <abe@cloudera.com> wrote:

> Anthony,
>
> Weird, that should be com.cloudera.sqoop.manager.ConnMananger. I suspect
> there's a problem with your sqoop-site.xml (if its even being parsed at
> all) or classpath. The sqoop startup script sets SQOOP_HOME_DIR,
> SQOOP_CONF_DIR, HADOOP_CLASSPATH, and a few other environment variables. I
> suspect there's something required there.
>
> -Abe
>
>
> On Wed, Jun 19, 2013 at 6:53 PM, corbacho anthony <acorbacho@nflabs.com>wrote:
>
>> I tried an same issues.
>> The problem is Sqoop cannot find com.cloudera.sqoop.apache.ConnMananger...
>>
>>
>> On Thu, Jun 20, 2013 at 10:33 AM, corbacho anthony <acorbacho@nflabs.com>wrote:
>>
>>> I used
>>> https://repository.cloudera.com/artifactory/cloudera-repos/com/cloudera/sqoop/sqoop/
>>>
>>> I will try with sqoop 1.4.3.
>>> Thank you Jarec.
>>>
>>> Anthony
>>>
>>>
>>> On Thu, Jun 20, 2013 at 10:28 AM, Jarek Jarcec Cecho <jarcec@apache.org>wrote:
>>>
>>>> Hi Anthony,
>>>> what maven repository are you using?
>>>>
>>>> Cloudera maven repository [1] seems to be containing all released
>>>> versions (instructions how to use this repository are in installation guide
>>>> [3]). Also thanks to SQOOP-1007 we do have vanila Sqoop artifacts available
>>>> in main maven repository [2].
>>>>
>>>> Jarcec
>>>>
>>>> Links:
>>>> 1:
>>>> https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/sqoop/sqoop/
>>>> 2: http://search.maven.org/#browse%7C585117616
>>>> 3:
>>>> http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Installation-Guide/cdh4ig_topic_31.html
>>>>
>>>> On Thu, Jun 20, 2013 at 10:17:35AM +0900, corbacho anthony wrote:
>>>> > hello Jarec and Abe.
>>>> >
>>>> > Abe: the debugging mode show nothing :(
>>>> > Jarec: In cloudera repo I can only find Sqoop 1.4.0.
>>>> >
>>>> > There is the full stack (in case):
>>>> > 2013-06-20 10:10:29,022 | ERROR | qtp1742391956-59 | ConnFactory |
>>>> Could
>>>> > not load ManagerFactory com.cloudera.sqoop.apache.ConnMananger (not
>>>> found)
>>>> > 2013-06-20 10:10:29,023 | ERROR | qtp1742391956-59 | BaseSqoopTool |
>>>> Got
>>>> > error creating database manager: java.io.IOException: No manager for
>>>> > connect string: jdbc:mysql://xxx:0000/x
>>>> > at org.apache.sqoop.ConnFactory.getManager(ConnFactory.java:119)
>>>> > at org.apache.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:202)
>>>> > at org.apache.sqoop.tool.ImportTool.init(ImportTool.java:83)
>>>> > at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:441)
>>>> > at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
>>>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>>> > at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
>>>> > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
>>>> > at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:49)
>>>> > at
>>>> com.nflabs.com.peloton.imprt.db.SqoopImport.tested(SqoopImport.java:131)
>>>> > at
>>>> com.nflabs.peloton.imprt.impl.ImprtSvcImpl.test(ImprtSvcImpl.java:191)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> org.apache.aries.proxy.impl.ProxyHandler$1.invoke(ProxyHandler.java:50)
>>>> > at
>>>> org.apache.aries.proxy.impl.DefaultWrapper.invoke(DefaultWrapper.java:31)
>>>> > at
>>>> org.apache.aries.proxy.impl.ProxyHandler.invoke(ProxyHandler.java:78)
>>>> > at com.sun.proxy.$Proxy76.test(Unknown Source)
>>>> > at
>>>> >
>>>> com.nflabs.peloton.rs.imprt.ImprtService.BrowseDirectory1(ImprtService.java:478)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:173)
>>>> > at
>>>> >
>>>> org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:89)
>>>> > at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:165)
>>>> > at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92)
>>>> > at
>>>> >
>>>> org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)
>>>> > at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
>>>> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>> > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>> > at
>>>> >
>>>> org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)
>>>> > at
>>>> >
>>>> org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:106)
>>>> > at
>>>> >
>>>> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:263)
>>>> > at
>>>> >
>>>> org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
>>>> > at
>>>> >
>>>> org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:207)
>>>> > at
>>>> >
>>>> org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:209)
>>>> > at
>>>> >
>>>> org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:152)
>>>> > at
>>>> >
>>>> org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:114)
>>>> > at
>>>> >
>>>> org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:185)
>>>> > at
>>>> >
>>>> org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:113)
>>>> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:693)
>>>> > at
>>>> >
>>>> org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:164)
>>>> > at
>>>> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:538)
>>>> > at
>>>> >
>>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:478)
>>>> > at
>>>> >
>>>> org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:70)
>>>> > at
>>>> >
>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
>>>> > at
>>>> >
>>>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:480)
>>>> > at
>>>> >
>>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225)
>>>> > at
>>>> >
>>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:937)
>>>> > at
>>>> >
>>>> org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:116)
>>>> > at
>>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
>>>> > at
>>>> >
>>>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
>>>> > at
>>>> >
>>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:871)
>>>> > at
>>>> >
>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
>>>> > at
>>>> >
>>>> org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:72)
>>>> > at
>>>> >
>>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:110)
>>>> > at org.eclipse.jetty.server.Server.handle(Server.java:346)
>>>> > at
>>>> >
>>>> org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:589)
>>>> > at
>>>> >
>>>> org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1048)
>>>> > at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:601)
>>>> > at
>>>> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:214)
>>>> > at
>>>> org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:411)
>>>> > at
>>>> >
>>>> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:535)
>>>> > at
>>>> >
>>>> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:40)
>>>> > at
>>>> >
>>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
>>>> > at java.lang.Thread.run(Thread.java:680)
>>>> >
>>>> > Thanks
>>>> > Anthony
>>>> >
>>>> >
>>>> > On Thu, Jun 20, 2013 at 2:52 AM, Abraham Elmahrek <abe@cloudera.com>
>>>> wrote:
>>>> >
>>>> > > Anthony,
>>>> > >
>>>> > > You could also try enabling debug logging to see what managers
are
>>>> being
>>>> > > loaded (if any). This can normally be done by adding "--verbose"
to
>>>> your
>>>> > > list of arguments. I'm not sure how it will work with your intended
>>>> use
>>>> > > case.
>>>> > >
>>>> > > -Abe
>>>> > >
>>>> > >
>>>> > > On Wed, Jun 19, 2013 at 7:57 AM, Jarek Jarcec Cecho <
>>>> jarcec@apache.org>wrote:
>>>> > >
>>>> > >> Hi Anthony,
>>>> > >> I'm glad to see your interested in Sqoop project! I have to
>>>> mention that
>>>> > >> whereas the direct usage of the Sqoop classes might work, it's
not
>>>> > >> officially supported API and you might hit some issues along
the
>>>> way.
>>>> > >>
>>>> > >> I would strongly suggest to upgrade to latest Sqoop version
as the
>>>> 1.4.0
>>>> > >> is quite old. In most cases using the --driver parameter is
not
>>>> needed as
>>>> > >> it will instruct Sqoop to use the Generic JDBC Driver instead
of
>>>> > >> specialized MySQL connector, I would suggest to drop this parameter
>>>> > >> entirely.
>>>> > >>
>>>> > >> Jarcec
>>>> > >>
>>>> > >> On Wed, Jun 19, 2013 at 03:25:51PM +0900, corbacho anthony
wrote:
>>>> > >> > Hello,
>>>> > >> >
>>>> > >> > I am trying to use Sqoop in an osgi pluging but it wont
work and
>>>> I dont
>>>> > >> > know why.
>>>> > >> > I keep getting the following error:
>>>> > >> > "2013-06-19 13:53:27,556 | ERROR | qtp122715895-276 |
>>>> BaseSqoopTool  |
>>>> > >> ?  ?
>>>> > >> > |  -  -  | Got error creating database manager:
>>>> java.io.IOException: No
>>>> > >> > manager for connect string: jdbc:mysql://...."
>>>> > >> >
>>>> > >> > I have include sqoop in my pom (1.4.0-cdh4b1) and I also
have my
>>>> mysql
>>>> > >> > driver loader (i can use in the code).
>>>> > >> >
>>>> > >> > the part who throw an exception is:
>>>> > >> >
>>>> > >> > String[] str = {
>>>> > >> > "import",
>>>> > >> > "--driver", "com.mysql.jdbc.Driver",
>>>> > >> > "--connect", "jdbc:mysql://___MY_DB___",
>>>> > >> > "--hadoop-home", __HOME__,
>>>> > >> > "--query", "Select [.....] AND $CONDITIONS",
>>>> > >> > "--split-by", "id",
>>>> > >> > "--target-dir", "hdfs://__SERVER__/__FOLDER__",
>>>> > >> > "--username", "foo",
>>>> > >> > "--password", "barou"};
>>>> > >> >
>>>> > >> > return Sqoop.runTool(str);
>>>> > >> >
>>>> > >> > Thanks,
>>>> > >> > Anthony
>>>> > >>
>>>> > >
>>>> > >
>>>>
>>>
>>>
>>
>

Mime
View raw message