sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abraham Elmahrek <...@cloudera.com>
Subject Re: Final Jiras and Release Date
Date Wed, 23 Jul 2014 17:38:50 GMT
Hey guys,

As soon as the Ivy and Avro version changes are in, let's start the
branching process?

-Abe


On Wed, Jul 23, 2014 at 1:02 AM, Venkat Ranganathan <
vranganathan@hortonworks.com> wrote:

>  Generally  the number of tasks is a hint to InputFormat.getSplits()
> based on number of splits wanted and it can potentially return
> different number of splits.
>
> If you see  ExportInputFormat, the split size is determined by
> combined file sizes divided by requested num mappers.   This is a
> integer division and  there is a potential for getting additional
> splits.
>
> Thanks
>
> Venkat
>
> On Tue, Jul 22, 2014 at 11:41 PM, David Robson
> <David.Robson@software.dell.com> wrote:
> > Yes OraOop is not enabled by default - and also this particular issue is
> only when using partitioning - so they would have turned on some extra
> flags. Then they also have to be using 1 more mapper than requested as you
> said - so it's a pretty specific combination of events - so I don't think
> we need to hold up the release for it.
> >
> > Do you recall what the behaviour should be in regards to this - the
> documentation leads me to believe if we request 4 mappers we should get 4
> mappers? If this is only a request and we may get more - perhaps we should
> update the documentation about this. Otherwise we could change the code to
> guarantee no more than 4 mappers?
> >
> > -----Original Message-----
> > From: Abraham Elmahrek [mailto:abe@cloudera.com]
> > Sent: Wednesday, 23 July 2014 3:16 PM
> > To: dev@sqoop.apache.org
> > Subject: Re: Final Jiras and Release Date
> >
> > Hey guys,
> >
> > AFAIK we can fall back onto the original connector? With that being
> said, OraOop is an awesome connector... So i'm completely open to waiting
> for a fix if it isn't too large.
> >
> > David, I'm assuming you're referring to the "-m" option? I do recall
> some cases where there may be one more split than desired if splitting is
> not clean. Hopefully Venkat has more insight!
> >
> > -Abe
> >
> >
> > On Tue, Jul 22, 2014 at 9:42 PM, David Robson <
> David.Robson@software.dell.com> wrote:
> >
> >> Hey Venkat,
> >>
> >> I had a look into that issue - it seems the problem is even though you
> >> request 4 mappers - Sqoop runs 5 mappers. Is the number of mappers
> >> meant to be guaranteed? For example if I say 4 mappers, should I get 4
> >> mappers? I guess there could be potential to get less mappers if the
> >> data was for example 1 row - then that would only be processed by 1
> >> mapper. But should it be possible to get more mappers than requested?
> >>
> >> OraOop is assuming there will be no more mappers than requested - so
> >> if this is a valid scenario we would need to modify OraOop. On the
> >> other hand if this is unexpected behaviour then we should look at why
> >> the num mappers is not working?
> >>
> >> Either way I don't think we need to fix it for this release - as it
> >> seems customers would be unlikely to hit this issue.
> >>
> >> David
> >>
> >> -----Original Message-----
> >> From: Venkat Ranganathan [mailto:vranganathan@hortonworks.com]
> >> Sent: Wednesday, 23 July 2014 1:04 PM
> >> To: dev@sqoop.apache.org
> >> Subject: Re: Final Jiras and Release Date
> >>
> >> Abe
> >>
> >> Do we want to target SQOOP-1388 also for 1.4.5 - the one that Vidya
> >> has raised for one of the Oracle connector export failure?
> >>
> >> My avro patch is in RB.
> >>
> >> Thanks
> >>
> >> Venkat
> >>
> >> On Tue, Jul 22, 2014 at 6:12 PM, Abraham Elmahrek <abe@cloudera.com>
> >> wrote:
> >> > Hey guys,
> >> >
> >> > It looks like we still have a couple of Jiras still open. Let's see
> >> > how things look tomorrow, but let's have these be the last two Jiras
> >> > slotted for this release.
> >> >
> >> > -Abe
> >> >
> >> >
> >> > On Fri, Jul 18, 2014 at 1:20 PM, Venkat Ranganathan <
> >> > vranganathan@hortonworks.com> wrote:
> >> >
> >> >> Sounds good.   I have already uploaded the patch for SQOOP-1358.  
It
> >> >> needs to be reviewed by a committer and then committed if no
> >> >> further changes are needed (or more work needed otherwise).
> >> >>
> >> >> Thanks for driving the release!
> >> >>
> >> >> Venkat
> >> >>
> >> >> On Fri, Jul 18, 2014 at 10:58 AM, Jarek Jarcec Cecho
> >> >> <jarcec@apache.org>
> >> >> wrote:
> >> >> > Seems reasonable to me, thank you for driving the release Abe!
> >> >> >
> >> >> > Jarcec
> >> >> >
> >> >> > On Jul 18, 2014, at 10:40 AM, Abraham Elmahrek <abe@cloudera.com>
> >> wrote:
> >> >> >
> >> >> >> Hey guys,
> >> >> >>
> >> >> >> I haven't seen a -1 on this suggestion. So let's create a
> >> >> >> release branch come Wednesday July 23rd.
> >> >> >>
> >> >> >> -Abe
> >> >> >>
> >> >> >>
> >> >> >> On Wed, Jul 16, 2014 at 11:38 AM, Abraham Elmahrek
> >> >> >> <abe@cloudera.com>
> >> >> wrote:
> >> >> >>
> >> >> >>> Are there any objections to this branch date?
> >> >> >>>
> >> >> >>> -Abe
> >> >> >>>
> >> >> >>>
> >> >> >>> On Mon, Jul 14, 2014 at 1:22 PM, Abraham Elmahrek
> >> >> >>> <abe@apache.org>
> >> >> wrote:
> >> >> >>>
> >> >> >>>> Hey guys,
> >> >> >>>>
> >> >> >>>> It looks like there are a couple of Jiras that are
in progress
> >> >> >>>> and slotted for the 1.4.5 release. If we aim for July
23rd to
> >> >> >>>> create a
> >> >> release
> >> >> >>>> branch, does that give every one enough time to finish
up what
> >> >> >>>> they're working on?
> >> >> >>>>
> >> >> >>>> See https://issues.apache.org/jira/browse/SQOOP-1353
for more
> >> >> >>>> information.
> >> >> >>>>
> >> >> >>>> -Abe
> >> >> >>>>
> >> >> >>>
> >> >> >>>
> >> >> >
> >> >>
> >> >> --
> >> >> CONFIDENTIALITY NOTICE
> >> >> NOTICE: This message is intended for the use of the individual or
> >> >> entity to which it is addressed and may contain information that is
> >> >> confidential, privileged and exempt from disclosure under
> >> >> applicable law. If the reader of this message is not the intended
> >> >> recipient, you are hereby notified that any printing, copying,
> >> >> dissemination, distribution, disclosure or forwarding of this
> >> >> communication is strictly prohibited. If you have received this
> >> >> communication in error, please contact the sender immediately and
> >> >> delete it from your
> >> system. Thank You.
> >> >>
> >>
> >> --
> >> CONFIDENTIALITY NOTICE
> >> NOTICE: This message is intended for the use of the individual or
> >> entity to which it is addressed and may contain information that is
> >> confidential, privileged and exempt from disclosure under applicable
> >> law. If the reader of this message is not the intended recipient, you
> >> are hereby notified that any printing, copying, dissemination,
> >> distribution, disclosure or forwarding of this communication is
> >> strictly prohibited. If you have received this communication in error,
> >> please contact the sender immediately and delete it from your system.
> Thank You.
> >>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message