sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abraham Elmahrek <...@cloudera.com>
Subject Re: Final Jiras and Release Date
Date Wed, 23 Jul 2014 20:19:57 GMT
Awesome! Thanks for your hard work guys!

Jarcec, can we use
https://github.com/apache/sqoop/commit/81624ddf3c8ca5834ab015ebafc8b8649ac36ab7?
In terms of when... let's shoot for 2PM PST?

-Abe


On Wed, Jul 23, 2014 at 12:32 PM, Jarek Jarcec Cecho <jarcec@apache.org>
wrote:

> I think that both are in. Do let me know when and from which commit you
> want me to branch Abe :-)
>
> Jarcec
>
> On Jul 23, 2014, at 10:38 AM, Abraham Elmahrek <abe@cloudera.com> wrote:
>
> > Hey guys,
> >
> > As soon as the Ivy and Avro version changes are in, let's start the
> > branching process?
> >
> > -Abe
> >
> >
> > On Wed, Jul 23, 2014 at 1:02 AM, Venkat Ranganathan <
> > vranganathan@hortonworks.com> wrote:
> >
> >> Generally  the number of tasks is a hint to InputFormat.getSplits()
> >> based on number of splits wanted and it can potentially return
> >> different number of splits.
> >>
> >> If you see  ExportInputFormat, the split size is determined by
> >> combined file sizes divided by requested num mappers.   This is a
> >> integer division and  there is a potential for getting additional
> >> splits.
> >>
> >> Thanks
> >>
> >> Venkat
> >>
> >> On Tue, Jul 22, 2014 at 11:41 PM, David Robson
> >> <David.Robson@software.dell.com> wrote:
> >>> Yes OraOop is not enabled by default - and also this particular issue
> is
> >> only when using partitioning - so they would have turned on some extra
> >> flags. Then they also have to be using 1 more mapper than requested as
> you
> >> said - so it's a pretty specific combination of events - so I don't
> think
> >> we need to hold up the release for it.
> >>>
> >>> Do you recall what the behaviour should be in regards to this - the
> >> documentation leads me to believe if we request 4 mappers we should get
> 4
> >> mappers? If this is only a request and we may get more - perhaps we
> should
> >> update the documentation about this. Otherwise we could change the code
> to
> >> guarantee no more than 4 mappers?
> >>>
> >>> -----Original Message-----
> >>> From: Abraham Elmahrek [mailto:abe@cloudera.com]
> >>> Sent: Wednesday, 23 July 2014 3:16 PM
> >>> To: dev@sqoop.apache.org
> >>> Subject: Re: Final Jiras and Release Date
> >>>
> >>> Hey guys,
> >>>
> >>> AFAIK we can fall back onto the original connector? With that being
> >> said, OraOop is an awesome connector... So i'm completely open to
> waiting
> >> for a fix if it isn't too large.
> >>>
> >>> David, I'm assuming you're referring to the "-m" option? I do recall
> >> some cases where there may be one more split than desired if splitting
> is
> >> not clean. Hopefully Venkat has more insight!
> >>>
> >>> -Abe
> >>>
> >>>
> >>> On Tue, Jul 22, 2014 at 9:42 PM, David Robson <
> >> David.Robson@software.dell.com> wrote:
> >>>
> >>>> Hey Venkat,
> >>>>
> >>>> I had a look into that issue - it seems the problem is even though you
> >>>> request 4 mappers - Sqoop runs 5 mappers. Is the number of mappers
> >>>> meant to be guaranteed? For example if I say 4 mappers, should I get
4
> >>>> mappers? I guess there could be potential to get less mappers if the
> >>>> data was for example 1 row - then that would only be processed by 1
> >>>> mapper. But should it be possible to get more mappers than requested?
> >>>>
> >>>> OraOop is assuming there will be no more mappers than requested - so
> >>>> if this is a valid scenario we would need to modify OraOop. On the
> >>>> other hand if this is unexpected behaviour then we should look at why
> >>>> the num mappers is not working?
> >>>>
> >>>> Either way I don't think we need to fix it for this release - as it
> >>>> seems customers would be unlikely to hit this issue.
> >>>>
> >>>> David
> >>>>
> >>>> -----Original Message-----
> >>>> From: Venkat Ranganathan [mailto:vranganathan@hortonworks.com]
> >>>> Sent: Wednesday, 23 July 2014 1:04 PM
> >>>> To: dev@sqoop.apache.org
> >>>> Subject: Re: Final Jiras and Release Date
> >>>>
> >>>> Abe
> >>>>
> >>>> Do we want to target SQOOP-1388 also for 1.4.5 - the one that Vidya
> >>>> has raised for one of the Oracle connector export failure?
> >>>>
> >>>> My avro patch is in RB.
> >>>>
> >>>> Thanks
> >>>>
> >>>> Venkat
> >>>>
> >>>> On Tue, Jul 22, 2014 at 6:12 PM, Abraham Elmahrek <abe@cloudera.com>
> >>>> wrote:
> >>>>> Hey guys,
> >>>>>
> >>>>> It looks like we still have a couple of Jiras still open. Let's
see
> >>>>> how things look tomorrow, but let's have these be the last two Jiras
> >>>>> slotted for this release.
> >>>>>
> >>>>> -Abe
> >>>>>
> >>>>>
> >>>>> On Fri, Jul 18, 2014 at 1:20 PM, Venkat Ranganathan <
> >>>>> vranganathan@hortonworks.com> wrote:
> >>>>>
> >>>>>> Sounds good.   I have already uploaded the patch for SQOOP-1358.
> It
> >>>>>> needs to be reviewed by a committer and then committed if no
> >>>>>> further changes are needed (or more work needed otherwise).
> >>>>>>
> >>>>>> Thanks for driving the release!
> >>>>>>
> >>>>>> Venkat
> >>>>>>
> >>>>>> On Fri, Jul 18, 2014 at 10:58 AM, Jarek Jarcec Cecho
> >>>>>> <jarcec@apache.org>
> >>>>>> wrote:
> >>>>>>> Seems reasonable to me, thank you for driving the release
Abe!
> >>>>>>>
> >>>>>>> Jarcec
> >>>>>>>
> >>>>>>> On Jul 18, 2014, at 10:40 AM, Abraham Elmahrek <abe@cloudera.com>
> >>>> wrote:
> >>>>>>>
> >>>>>>>> Hey guys,
> >>>>>>>>
> >>>>>>>> I haven't seen a -1 on this suggestion. So let's create
a
> >>>>>>>> release branch come Wednesday July 23rd.
> >>>>>>>>
> >>>>>>>> -Abe
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Wed, Jul 16, 2014 at 11:38 AM, Abraham Elmahrek
> >>>>>>>> <abe@cloudera.com>
> >>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Are there any objections to this branch date?
> >>>>>>>>>
> >>>>>>>>> -Abe
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Mon, Jul 14, 2014 at 1:22 PM, Abraham Elmahrek
> >>>>>>>>> <abe@apache.org>
> >>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hey guys,
> >>>>>>>>>>
> >>>>>>>>>> It looks like there are a couple of Jiras that
are in progress
> >>>>>>>>>> and slotted for the 1.4.5 release. If we aim
for July 23rd to
> >>>>>>>>>> create a
> >>>>>> release
> >>>>>>>>>> branch, does that give every one enough time
to finish up what
> >>>>>>>>>> they're working on?
> >>>>>>>>>>
> >>>>>>>>>> See https://issues.apache.org/jira/browse/SQOOP-1353
for more
> >>>>>>>>>> information.
> >>>>>>>>>>
> >>>>>>>>>> -Abe
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> CONFIDENTIALITY NOTICE
> >>>>>> NOTICE: This message is intended for the use of the individual
or
> >>>>>> entity to which it is addressed and may contain information
that is
> >>>>>> confidential, privileged and exempt from disclosure under
> >>>>>> applicable law. If the reader of this message is not the intended
> >>>>>> recipient, you are hereby notified that any printing, copying,
> >>>>>> dissemination, distribution, disclosure or forwarding of this
> >>>>>> communication is strictly prohibited. If you have received this
> >>>>>> communication in error, please contact the sender immediately
and
> >>>>>> delete it from your
> >>>> system. Thank You.
> >>>>>>
> >>>>
> >>>> --
> >>>> CONFIDENTIALITY NOTICE
> >>>> NOTICE: This message is intended for the use of the individual or
> >>>> entity to which it is addressed and may contain information that is
> >>>> confidential, privileged and exempt from disclosure under applicable
> >>>> law. If the reader of this message is not the intended recipient, you
> >>>> are hereby notified that any printing, copying, dissemination,
> >>>> distribution, disclosure or forwarding of this communication is
> >>>> strictly prohibited. If you have received this communication in error,
> >>>> please contact the sender immediately and delete it from your system.
> >> Thank You.
> >>>>
> >>
> >> --
> >> CONFIDENTIALITY NOTICE
> >> NOTICE: This message is intended for the use of the individual or
> entity to
> >> which it is addressed and may contain information that is confidential,
> >> privileged and exempt from disclosure under applicable law. If the
> reader
> >> of this message is not the intended recipient, you are hereby notified
> that
> >> any printing, copying, dissemination, distribution, disclosure or
> >> forwarding of this communication is strictly prohibited. If you have
> >> received this communication in error, please contact the sender
> immediately
> >> and delete it from your system. Thank You.
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message