sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Jarcec Cecho <jar...@apache.org>
Subject Re: How to invoke extractor during IMPORT Job Submition process- Sqoop version 1.99.1 Connector
Date Tue, 22 Oct 2013 15:35:59 GMT
Hi Kunal,
the extractor is connector driver rather then user given. As a result you can't specify which
extractor will be used as this is specified by the connector. You will have to create your
own connector if you need to call custom extractor.

Jarcec

On Thu, Oct 17, 2013 at 03:14:16PM +0530, K Singh wrote:
> Hi ,
> 
> Thanks for clarification.
> 
> I have already followed below URL to make my own generic connector :
> 
> https://github.com/apache/sqoop/blob/sqoop2/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java#L38
> 
> 
> But the requirement is, once I have my own connector which is working fine,
> I want to invoke Extractor during IMPORT job. The built in generic
> connector also not calling Extractor. So Is there any way to invoke
> extractor or how and when Extractor is being called?
> 
> In simple words, If we already developed/built generic connector and
> working fine then How can Import job submission/I call or invoke extractor
> class ( e.q. GenericJdbcImportExtractor.class,) during IMPORT process. I
> want to just call this class default method.
> 
> Please guide me for this. Is there any additional command line arguments or
> something like which we can use to invoke/call extractor  because by
> default it is not being called in IMPORT job. Like we have for validation:
> 
> *Validation arguments More
> Details<http://archive.cloudera.com/cdh4/cdh/4/sqoop-1.4.3-cdh4.4.0/SqoopUserGuide.html#validation>
> *
> Argument Description --validate Enable validation of data copied, supports
> single table copy only. --validator <class-name> Specify validator class to
> use. --validation-threshold <class-name> Specify validation threshold class
> to use. +--validation-failurehandler <class-name >+ Specify validation
> failure handler class to use.
> 
> 
> So is there anything for extractor as well?
> 
> Thanks.
> 
> 
> 
> 
> On Fri, Oct 11, 2013 at 9:13 PM, Jarek Jarcec Cecho <jarcec@apache.org>wrote:
> 
> > Hi Kunal,
> > Sqoop2 is currently under heavy development, so we do not have a "How to
> > create connector" guide available at the moment. We will definitely have
> > one once the interfaces will settle down.
> >
> > For the time being, I'm afraid that you will need to follow existing code.
> > If you need to call custom import extractor in your new connector, you will
> > need to specify it in the Importer structure that is exposed by your
> > connector. Please take a look on the Generic JDBC Connector example:
> >
> >
> > https://github.com/apache/sqoop/blob/sqoop2/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java#L38
> >
> > Jarcec
> >
> > On Wed, Oct 09, 2013 at 06:31:22PM +0530, Kunal Sachdeva wrote:
> > > Hi,
> > >
> > > *Subject: Want to invoke custom extractor class run method for printing
> > > data- MySql table's columns values data in Import job*.
> > >
> > > I am newby to Sqoop2 (*Currently working on connector version 1.99.1*)
> > and
> > > want some help from you. I have developed Sqoop2 connector similar to
> > > generic connector 1.99.1 provided by apache, here
> > > http://sqoop.apache.org/docs/1.99.1/
> > > .
> > >
> > > In other words my connector is 99% similar to generic connector 1.99.1
> > but
> > > have different name for packages and classes.I have tested with my own
> > > connector  Import and Export both are working fine. But I want to print
> > > data during Import job submission and before it writes data to
> > > given/specific file, I mean when we import MySql Database table and
> > submit
> > > the Import job then whatever data Sqoop2 is importing from that table, I
> > > want all rows,columns to be appended with my name or some other value
> > 'abc'
> > > through connector.
> > >
> > > I know, Each connector in Sqoop2 extends class SqoopConnector by
> > providing
> > > classes that should be called. If I need to run custom extractor, I will
> > > need to create custom connector that will have different extractor. On
> > the
> > > client side, I will have to create connection for your new connector
> > > instead of the build-in Generic JDBC Connector.
> > >
> > > I have already done these above steps and My connector is working fine
> > for
> > > both Import,Export jobs and I have seen result in desired location.
> > >
> > > But, as said in Import job, to append anything in table rows data before
> > > writing them to any file , we have to code in extractor. My extractor is
> > > 100% same as generic connector extractor. But when and how to invoke this
> > > custom conector extractor is the main thing which I do not know and I
> > want
> > > to invoke this.
> > >
> > > Please help me which class and method I need to add/modify so that I can
> > > get invoke my connector extractor.
> > >
> > > I have followed all steps mentioned here
> > > http://sqoop.apache.org/docs/1.99.1/ but invoking extractor is hard
> > thing
> > > for me. I do not know which value I have to give in command prompt and
> > all.
> > >
> > > Thanks in advance.
> > >
> > > --
> > > Regards,
> > > Kunal
> >

Mime
View raw message