sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Jarcec Cecho <jar...@apache.org>
Subject Re: Import failure due to thrift incompatibility?
Date Fri, 11 Oct 2013 15:36:46 GMT
Hi lars,
those packages are in build directory that is not part of the distribution tarball. In case
that you are compiling Sqoop yourself, I would suggest to call target "ant tar" and distribute
the generated tarball from build directory to your cluster rather than copying entire working
directory.

Jarcec

On Fri, Oct 11, 2013 at 03:56:45PM +1100, Lars Yencken wrote:
> Hi Jarek,
> 
> I'm not using HBase on that cluster, which I've verified by hunting for
> binaries. Sqoop seems to be pulling it in though:
> 
> hadoop@ip-XXX-XXX-XXX-XXX:~/sqoop$ find . -iname '*hbase*.jar'
> ./build/ivy/lib/sqoop/hadoop100test/hbase-0.92.0-tests.jar
> ./build/ivy/lib/sqoop/hadoop100test/hbase-0.92.0.jar
> 
> Any idea if I can prune it from Sqoop's build?
> 
> Cheers,
> Lars
> 
> 
> On 11 October 2013 02:54, Jarek Jarcec Cecho <jarcec@apache.org> wrote:
> 
> > Hi Lars,
> > I've seen this issue many times and I can confirm your findings - this
> > exception is most likely being thrown because multiple incompatible thrift
> > implementations are available on Sqoop classpath.
> >
> > However Sqoop itself do not ships with thrift as we are not using thrift.
> > Usually it's the other dependent projects that are bringing thrift as
> > transitive dependency. Are you by any chance also using HBase? HBase is
> > known to ship it's own thrift version that is usually not the same as the
> > one from Hive.
> >
> > Jarcec
> >
> > On Tue, Oct 08, 2013 at 05:26:52PM +1100, Lars Yencken wrote:
> > > Hi all,
> > >
> > > I'm having trouble with what should be a standard Sqoop import job, from
> > > MySQL to Hive. On a new cluster, I get this exception when I do my import
> > > run:
> > >
> > > java.lang.NoSuchMethodError: org.apache.thrift.EncodingUtils.setBit(BIZ)B
> > >
> > > Full stack trace: https://gist.github.com/larsyencken/6879918
> > > Environment: Sqoop 1.4.4 / Hive 0.8.1.8 / Hadoop 1.0.3 on EMR
> > >
> > > The problem only occurs with the Hive import. If I take it out and just
> > > import from MySQL to HDFS, the data comes through without problems.
> > >
> > > I have a feeling the problem is to do with multiple incompatible thrift
> > > versions (Hive has libthrift-0.7.0.jar, Sqoop uses libthrift-0.9.0.jar).
> > >
> > > Any suggestions on getting these two to play nicely together?
> > >
> > > Best,
> > > Lars
> >

Mime
View raw message