hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Laurent H <laurent.hat...@gmail.com>
Subject Re: Error while loading bulk data from pig to hbase
Date Fri, 20 Nov 2015 17:08:53 GMT
You can develop a bulk load in Java :
http://www.deerwalk.com/blog/bulk-importing-data/ ==> reducers are the
number of regions (reducers is, physically, HFile) in your Hbase and bulk
mode doesn't use Memstore.
You can use Flume but i don't think that is an great idea to use bulk mode
with Flume... (you can use it with normal loading - which use Memstore)




--
Laurent HATIER - Consultant Big Data & Business Intelligence chez CapGemini
fr.linkedin.com/pub/laurent-hatier/25/36b/a86/
<http://fr.linkedin.com/pub/laurent-h/25/36b/a86/>

2015-11-18 22:59 GMT+01:00 Ted Yu <yuzhihong@gmail.com>:

> See Daniel's comment on PIG-4728 .
>
> Cheers
>
> On Wed, Nov 18, 2015 at 1:46 AM, Heng Chen <heng.chen.1986@gmail.com>
> wrote:
>
> > org.apache.pig.backend.hadoop.hbase.HBaseStorage  is in pig project.
> >
> > *ERROR:pig script failed to validate: java.lang.RuntimeException: could
> not
> > instantiate 'org.apache.pig.backend.hadoop.hbase.HBaseStorage' with
> > arguments.*
> >
> > This message means the arguments is not correct.
> > Please check your argument format, is it right?
> >
> >
> > Thanks
> >
> > 2015-11-18 17:36 GMT+08:00 Amit Hora <hora.amit@gmail.com>:
> >
> > > If you are cool with using some other tool for bulk insertion you can
> go
> > > with Apache Flume.
> > >
> > > -----Original Message-----
> > > From: "Nishant Aggarwal" <nishant.agg@gmail.com>
> > > Sent: ‎18-‎11-‎2015 14:28
> > > To: "user@hbase.apache.org" <user@hbase.apache.org>
> > > Subject: Re: Error while loading bulk data from pig to hbase
> > >
> > > Dear All,
> > >
> > > Please help us on this. We need to bulk import data into Hbase using
> pig
> > > (or any alternate way).
> > >
> > > Any help on this will be appreciated.
> > >
> > > Thanks and Regards
> > > Nishant Aggarwal, PMP
> > > Cell No:- +91 99588 94305
> > > http://in.linkedin.com/pub/nishant-aggarwal/53/698/11b
> > >
> > >
> > > On Tue, Nov 17, 2015 at 4:10 AM, Laurent H <laurent.hatier@gmail.com>
> > > wrote:
> > >
> > > > I remember that Pig lib with HBaseStorage (0.13 or 0.14) doesn't
> accept
> > > > bulk loading, (if you look at the java class, you could see that
> there
> > is
> > > > only put method and no bulk function...) Hope it's available righ
> now !
> > > >
> > > > --
> > > > Laurent HATIER - Consultant Big Data & Business Intelligence chez
> > > CapGemini
> > > > fr.linkedin.com/pub/laurent-hatier/25/36b/a86/
> > > > <http://fr.linkedin.com/pub/laurent-h/25/36b/a86/>
> > > >
> > > > 2015-11-05 13:58 GMT+01:00 Naresh Reddy <
> > > naresh.reddy@aletheconsulting.com
> > > > >:
> > > >
> > > > > Hi
> > > > >
> > > > > I have already replaced the hbase version with
> > > "*hbase95.version=1.1.2*"
> > > > in
> > > > > libraries.properties file and compiled it, but I am getting the
> same
> > > > error.
> > > > >
> > > > > Regards
> > > > > Naresh
> > > > >
> > > > > On Wed, Nov 4, 2015 at 11:29 PM, Daniel Dai <daijyc@gmail.com>
> > wrote:
> > > > >
> > > > > > Will need to change ivy/libraries.properties, specify the right
> > hbase
> > > > > > version and compile again.
> > > > > >
> > > > > > On Wed, Nov 4, 2015 at 6:31 AM, Ted Yu <yuzhihong@gmail.com>
> > wrote:
> > > > > >
> > > > > > > ... 22 moreCaused by: java.lang.NoSuchMethodError:
> > > > > > > org.apache.hadoop.hbase.client.Scan.setCacheBlocks(Z)V
   at
> > > > > > >
> > > > > > > Looks like the version of Pig you use is not compiled against
> > hbase
> > > > > 1.1.2
> > > > > > >
> > > > > > > This is related:
> > > > > > > Author: Enis Soztutar <enis@apache.org>
> > > > > > > Date:   Fri Sep 5 18:48:38 2014 -0700
> > > > > > >
> > > > > > >     HBASE-10841 Scan,Get,Put,Delete,etc setters should
> > consistently
> > > > > > return
> > > > > > > this
> > > > > > >
> > > > > > > FYI
> > > > > > >
> > > > > > > On Tue, Nov 3, 2015 at 10:36 PM, Naresh Reddy <
> > > > > > > naresh.reddy@aletheconsulting.com> wrote:
> > > > > > >
> > > > > > >> Hi
> > > > > > >> Thanks for the reply.Below is the full error log.
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> *Pig Stack Trace---------------ERROR 1200: Pig script
failed
> to
> > > > parse:
> > > > > > >> <line 2, column 0> pig script failed to validate:
> > > > > > >> java.lang.RuntimeException: could not instantiate
> > > > > > >> 'org.apache.pig.backend.hadoop.hbase.HBaseStorage'
with
> > arguments
> > > > > > >> '[info:fname info:lname]'Failed to parse: Pig script
failed to
> > > > parse:
> > > > > > >> <line
> > > > > > >> 2, column 0> pig script failed to validate:
> > > > > java.lang.RuntimeException:
> > > > > > >> could not instantiate
> > > > > 'org.apache.pig.backend.hadoop.hbase.HBaseStorage'
> > > > > > >> with arguments '[info:fname info:lname]'    at
> > > > > > >>
> > > > > >
> > > >
> > org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:199)
> > > > > > >> at
> > > org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1707)
> > > > > > at
> > > > > > >>
> > org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1680)
> > > > > at
> > > > > > >> org.apache.pig.PigServer.registerQuery(PigServer.java:623)
> at
> > > > > > >>
> > > > >
> > >
> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:1063)
> > > > > > >> at
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:501)
> > > > > > >> at
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230)
> > > > > > >> at
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
> > > > > > >> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66)
   at
> > > > > > >> org.apache.pig.Main.run(Main.java:558)    at
> > > > > > >> org.apache.pig.Main.main(Main.java:170)    at
> > > > > > >> sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
> > at
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > > > > > >> at
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > > > >> at java.lang.reflect.Method.invoke(Method.java:606)
   at
> > > > > > >> org.apache.hadoop.util.RunJar.main(RunJar.java:212)Caused
by:
> > > <line
> > > > 2,
> > > > > > >> column 0> pig script failed to validate:
> > > java.lang.RuntimeException:
> > > > > > could
> > > > > > >> not instantiate
> > 'org.apache.pig.backend.hadoop.hbase.HBaseStorage'
> > > > > with
> > > > > > >> arguments '[info:fname info:lname]'    at
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.parser.LogicalPlanBuilder.buildStoreOp(LogicalPlanBuilder.java:1009)
> > > > > > >> at
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGenerator.java:7775)
> > > > > > >> at
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1669)
> > > > > > >> at
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
> > > > > > >> at
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
> > > > > > >> at
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
> > > > > > >> at
> > > > > > >>
> > > > > >
> > > >
> > org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:191)
> > > > > > >> ... 15 moreCaused by: java.lang.RuntimeException: could
not
> > > > > instantiate
> > > > > > >> 'org.apache.pig.backend.hadoop.hbase.HBaseStorage'
with
> > arguments
> > > > > > >> '[info:fname info:lname]'    at
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:772)
> > > > > > >> at
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.parser.LogicalPlanBuilder.buildStoreOp(LogicalPlanBuilder.java:988)
> > > > > > >> ... 21 moreCaused by:
> > java.lang.reflect.InvocationTargetException
> > > > > at
> > > > > > >> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > > > Method)
> > > > > > >> at
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> > > > > > >> at
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> > > > > > >> at
> > java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> > > > > at
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:740)
> > > > > > >> ... 22 moreCaused by: java.lang.NoSuchMethodError:
> > > > > > >> org.apache.hadoop.hbase.client.Scan.setCacheBlocks(Z)V
   at
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.backend.hadoop.hbase.HBaseStorage.initScan(HBaseStorage.java:405)
> > > > > > >> at
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.backend.hadoop.hbase.HBaseStorage.<init>(HBaseStorage.java:346)
> > > > > > >> at
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.backend.hadoop.hbase.HBaseStorage.<init>(HBaseStorage.java:235)
> > > > > > >> ... 27
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> more================================================================================*
> > > > > > >>
> > > > > > >> Regards
> > > > > > >> Naresh
> > > > > > >>
> > > > > > >> On Wed, Nov 4, 2015 at 11:46 AM, Naresh Reddy <
> > > > > > >> naresh.reddy@aletheconsulting.com> wrote:
> > > > > > >>
> > > > > > >> > Hi
> > > > > > >> > Thanks for the reply.PFA for the full error.
> > > > > > >> >
> > > > > > >> > Regard
> > > > > > >> > Naresh
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Wed, Nov 4, 2015 at 11:29 AM, Ted Yu <
> yuzhihong@gmail.com>
> > > > > wrote:
> > > > > > >> >
> > > > > > >> >> Naresh:
> > > > > > >> >> Can you pastebin the full error ?
> > > > > > >> >> It should be in pig_XXXX.log
> > > > > > >> >>
> > > > > > >> >> Cheers
> > > > > > >> >>
> > > > > > >> >> > On Nov 3, 2015, at 9:07 PM, Naresh Reddy
<
> > > > > > >> >> naresh.reddy@aletheconsulting.com> wrote:
> > > > > > >> >> >
> > > > > > >> >> > Hi
> > > > > > >> >> >
> > > > > > >> >> > I am getting the below error while loading
bulk data from
> > pig
> > > > to
> > > > > > >> hbase
> > > > > > >> >> > through HBaseStorage.Please help me to
resolve this
> > > > issue.Thanks
> > > > > in
> > > > > > >> >> advance.
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >> > *ERROR:pig script failed to validate:
> > > > java.lang.RuntimeException:
> > > > > > >> could
> > > > > > >> >> not
> > > > > > >> >> > instantiate
> > > 'org.apache.pig.backend.hadoop.hbase.HBaseStorage'
> > > > > with
> > > > > > >> >> > arguments.*
> > > > > > >> >> >
> > > > > > >> >> > Below is my pig script.
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >> > *register
> > > > > > '/home/hduser/hbase-1.1.2/lib/zookeeper-3.4.6.jar';register
> > > > > > >> >> > '/home/hduser/hbase-1.1.2/lib/guava-12.0.1.jar';register
> > > > > > >> >> >
> > > '/home/hduser/hbase-1.1.2/lib/hbase-common-1.1.2.jar';register
> > > > > > >> >> >
> > > '/home/hduser/hbase-1.1.2/lib/hbase-client-1.1.2.jar';register
> > > > > > >> >> >
> > > > '/home/hduser/hbase-1.1.2/lib/protobuf-java-2.5.0.jar';raw_data =
> > > > > > >> LOAD
> > > > > > >> >> > '/user/hduser/input.csv' USING PigStorage(',')
AS
> > > > (id:chararray,
> > > > > > >> >> > fname:chararray, lname:chararray);STORE
raw_data INTO
> > > > > > >> >> > 'hbase://sample_names' USING
> > > > > > >> >> > org.apache.pig.backend.hadoop.hbase.HBaseStorage
> > ('info:fname
> > > > > > >> >> info:lname');*
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >> > Regards
> > > > > > >> >> > Naresh
> > > > > > >> >>
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message