sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From K Singh <k.singh....@gmail.com>
Subject Re: How to invoke extractor during IMPORT Job Submition process- Sqoop version 1.99.1 Connector
Date Thu, 17 Oct 2013 09:44:16 GMT
Hi ,

Thanks for clarification.

I have already followed below URL to make my own generic connector :


But the requirement is, once I have my own connector which is working fine,
I want to invoke Extractor during IMPORT job. The built in generic
connector also not calling Extractor. So Is there any way to invoke
extractor or how and when Extractor is being called?

In simple words, If we already developed/built generic connector and
working fine then How can Import job submission/I call or invoke extractor
class ( e.q. GenericJdbcImportExtractor.class,) during IMPORT process. I
want to just call this class default method.

Please guide me for this. Is there any additional command line arguments or
something like which we can use to invoke/call extractor  because by
default it is not being called in IMPORT job. Like we have for validation:

*Validation arguments More
Argument Description --validate Enable validation of data copied, supports
single table copy only. --validator <class-name> Specify validator class to
use. --validation-threshold <class-name> Specify validation threshold class
to use. +--validation-failurehandler <class-name >+ Specify validation
failure handler class to use.

So is there anything for extractor as well?


On Fri, Oct 11, 2013 at 9:13 PM, Jarek Jarcec Cecho <jarcec@apache.org>wrote:

> Hi Kunal,
> Sqoop2 is currently under heavy development, so we do not have a "How to
> create connector" guide available at the moment. We will definitely have
> one once the interfaces will settle down.
> For the time being, I'm afraid that you will need to follow existing code.
> If you need to call custom import extractor in your new connector, you will
> need to specify it in the Importer structure that is exposed by your
> connector. Please take a look on the Generic JDBC Connector example:
> https://github.com/apache/sqoop/blob/sqoop2/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnector.java#L38
> Jarcec
> On Wed, Oct 09, 2013 at 06:31:22PM +0530, Kunal Sachdeva wrote:
> > Hi,
> >
> > *Subject: Want to invoke custom extractor class run method for printing
> > data- MySql table's columns values data in Import job*.
> >
> > I am newby to Sqoop2 (*Currently working on connector version 1.99.1*)
> and
> > want some help from you. I have developed Sqoop2 connector similar to
> > generic connector 1.99.1 provided by apache, here
> > http://sqoop.apache.org/docs/1.99.1/
> > .
> >
> > In other words my connector is 99% similar to generic connector 1.99.1
> but
> > have different name for packages and classes.I have tested with my own
> > connector  Import and Export both are working fine. But I want to print
> > data during Import job submission and before it writes data to
> > given/specific file, I mean when we import MySql Database table and
> submit
> > the Import job then whatever data Sqoop2 is importing from that table, I
> > want all rows,columns to be appended with my name or some other value
> 'abc'
> > through connector.
> >
> > I know, Each connector in Sqoop2 extends class SqoopConnector by
> providing
> > classes that should be called. If I need to run custom extractor, I will
> > need to create custom connector that will have different extractor. On
> the
> > client side, I will have to create connection for your new connector
> > instead of the build-in Generic JDBC Connector.
> >
> > I have already done these above steps and My connector is working fine
> for
> > both Import,Export jobs and I have seen result in desired location.
> >
> > But, as said in Import job, to append anything in table rows data before
> > writing them to any file , we have to code in extractor. My extractor is
> > 100% same as generic connector extractor. But when and how to invoke this
> > custom conector extractor is the main thing which I do not know and I
> want
> > to invoke this.
> >
> > Please help me which class and method I need to add/modify so that I can
> > get invoke my connector extractor.
> >
> > I have followed all steps mentioned here
> > http://sqoop.apache.org/docs/1.99.1/ but invoking extractor is hard
> thing
> > for me. I do not know which value I have to give in command prompt and
> all.
> >
> > Thanks in advance.
> >
> > --
> > Regards,
> > Kunal

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message