sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Warren <james.war...@stanfordalumni.org>
Subject Re: does sqoop support SQL_BUFFER_RESULT to help with long running queries?
Date Fri, 06 Jan 2012 02:35:12 GMT
Hi Arvind -

Thanks for the suggestions!  I tried the first option but it
fortunately failed.  From the logs:

12/01/05 18:24:56 INFO db.DataDrivenDBInputFormat: BoundingValsQuery:
SELECT MIN(id), MAX(id) FROM (SELECT SQL_BUFFER_RESULT id ......

and SQL_BUFFER_RESULT apparently only works for top-level select
queries.  Nice idea, though.

Looks like I'm left with option #2.  More work, but I like getting my
hands dirty.

cheers,
-James

On Thu, Jan 5, 2012 at 10:08 AM, Arvind Prabhakar <arvind@apache.org> wrote:
>
> Hi James,
>
> This has not been tested before - but you could possibly use the
> --query option to supply your own query that contains the necessary
> optimization hints. If you find that not working, or limiting in some
> way - you could also try extending the
> org.apache.sqoop.manager.MySQLManager implementation to create your
> own connector that does understand such optimizations.
>
> Thanks,
> Arvind
>
> On Wed, Jan 4, 2012 at 4:45 PM, James Warren
> <james.warren@stanfordalumni.org> wrote:
> > Greetings -
> >
> > Was wondering / hoping if it were possible to configure sqoop to use
> > SQL_BUFFER_RESULT in the generated select queries.  From my understanding
> > sqoop uses READ_COMMITTED as the isolation level, but I'd ideally like to
> > release the write locks as soon as possible to help alleviate any
> > replication lag.
> >
> > Many thanks,
> > -jw

Mime
View raw message