sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abraham Elmahrek <...@cloudera.com>
Subject Re: Final Jiras and Release Date
Date Wed, 23 Jul 2014 21:04:03 GMT
Agreed.

I'll dig into the test failures anyways. Thanks for your help man!


On Wed, Jul 23, 2014 at 2:00 PM, Jarek Jarcec Cecho <jarcec@apache.org>
wrote:

> Too late, I’m already in process of branching :-) But that’s fine - we can
> always cherry-pick whatever will be necessary.
>
> Jarcec
>
> On Jul 23, 2014, at 1:57 PM, Abraham Elmahrek <abe@cloudera.com> wrote:
>
> > Ah actually, it seems there is a broken OraOop test case. Let me see
> what's
> > going on there before we branch. Jarcec, will you be free around 4PM to
> > branch?
> >
> > -Abe
> >
> >
> > On Wed, Jul 23, 2014 at 1:19 PM, Abraham Elmahrek <abe@cloudera.com>
> wrote:
> >
> >> Awesome! Thanks for your hard work guys!
> >>
> >> Jarcec, can we use
> >>
> https://github.com/apache/sqoop/commit/81624ddf3c8ca5834ab015ebafc8b8649ac36ab7
> ?
> >> In terms of when... let's shoot for 2PM PST?
> >>
> >> -Abe
> >>
> >>
> >> On Wed, Jul 23, 2014 at 12:32 PM, Jarek Jarcec Cecho <jarcec@apache.org
> >
> >> wrote:
> >>
> >>> I think that both are in. Do let me know when and from which commit you
> >>> want me to branch Abe :-)
> >>>
> >>> Jarcec
> >>>
> >>> On Jul 23, 2014, at 10:38 AM, Abraham Elmahrek <abe@cloudera.com>
> wrote:
> >>>
> >>>> Hey guys,
> >>>>
> >>>> As soon as the Ivy and Avro version changes are in, let's start the
> >>>> branching process?
> >>>>
> >>>> -Abe
> >>>>
> >>>>
> >>>> On Wed, Jul 23, 2014 at 1:02 AM, Venkat Ranganathan <
> >>>> vranganathan@hortonworks.com> wrote:
> >>>>
> >>>>> Generally  the number of tasks is a hint to InputFormat.getSplits()
> >>>>> based on number of splits wanted and it can potentially return
> >>>>> different number of splits.
> >>>>>
> >>>>> If you see  ExportInputFormat, the split size is determined by
> >>>>> combined file sizes divided by requested num mappers.   This is
a
> >>>>> integer division and  there is a potential for getting additional
> >>>>> splits.
> >>>>>
> >>>>> Thanks
> >>>>>
> >>>>> Venkat
> >>>>>
> >>>>> On Tue, Jul 22, 2014 at 11:41 PM, David Robson
> >>>>> <David.Robson@software.dell.com> wrote:
> >>>>>> Yes OraOop is not enabled by default - and also this particular
> issue
> >>> is
> >>>>> only when using partitioning - so they would have turned on some
> extra
> >>>>> flags. Then they also have to be using 1 more mapper than requested
> as
> >>> you
> >>>>> said - so it's a pretty specific combination of events - so I don't
> >>> think
> >>>>> we need to hold up the release for it.
> >>>>>>
> >>>>>> Do you recall what the behaviour should be in regards to this
- the
> >>>>> documentation leads me to believe if we request 4 mappers we should
> >>> get 4
> >>>>> mappers? If this is only a request and we may get more - perhaps
we
> >>> should
> >>>>> update the documentation about this. Otherwise we could change the
> >>> code to
> >>>>> guarantee no more than 4 mappers?
> >>>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Abraham Elmahrek [mailto:abe@cloudera.com]
> >>>>>> Sent: Wednesday, 23 July 2014 3:16 PM
> >>>>>> To: dev@sqoop.apache.org
> >>>>>> Subject: Re: Final Jiras and Release Date
> >>>>>>
> >>>>>> Hey guys,
> >>>>>>
> >>>>>> AFAIK we can fall back onto the original connector? With that
being
> >>>>> said, OraOop is an awesome connector... So i'm completely open to
> >>> waiting
> >>>>> for a fix if it isn't too large.
> >>>>>>
> >>>>>> David, I'm assuming you're referring to the "-m" option? I do
recall
> >>>>> some cases where there may be one more split than desired if
> splitting
> >>> is
> >>>>> not clean. Hopefully Venkat has more insight!
> >>>>>>
> >>>>>> -Abe
> >>>>>>
> >>>>>>
> >>>>>> On Tue, Jul 22, 2014 at 9:42 PM, David Robson <
> >>>>> David.Robson@software.dell.com> wrote:
> >>>>>>
> >>>>>>> Hey Venkat,
> >>>>>>>
> >>>>>>> I had a look into that issue - it seems the problem is even
though
> >>> you
> >>>>>>> request 4 mappers - Sqoop runs 5 mappers. Is the number
of mappers
> >>>>>>> meant to be guaranteed? For example if I say 4 mappers,
should I
> get
> >>> 4
> >>>>>>> mappers? I guess there could be potential to get less mappers
if
> the
> >>>>>>> data was for example 1 row - then that would only be processed
by 1
> >>>>>>> mapper. But should it be possible to get more mappers than
> requested?
> >>>>>>>
> >>>>>>> OraOop is assuming there will be no more mappers than requested
-
> so
> >>>>>>> if this is a valid scenario we would need to modify OraOop.
On the
> >>>>>>> other hand if this is unexpected behaviour then we should
look at
> why
> >>>>>>> the num mappers is not working?
> >>>>>>>
> >>>>>>> Either way I don't think we need to fix it for this release
- as it
> >>>>>>> seems customers would be unlikely to hit this issue.
> >>>>>>>
> >>>>>>> David
> >>>>>>>
> >>>>>>> -----Original Message-----
> >>>>>>> From: Venkat Ranganathan [mailto:vranganathan@hortonworks.com]
> >>>>>>> Sent: Wednesday, 23 July 2014 1:04 PM
> >>>>>>> To: dev@sqoop.apache.org
> >>>>>>> Subject: Re: Final Jiras and Release Date
> >>>>>>>
> >>>>>>> Abe
> >>>>>>>
> >>>>>>> Do we want to target SQOOP-1388 also for 1.4.5 - the one
that Vidya
> >>>>>>> has raised for one of the Oracle connector export failure?
> >>>>>>>
> >>>>>>> My avro patch is in RB.
> >>>>>>>
> >>>>>>> Thanks
> >>>>>>>
> >>>>>>> Venkat
> >>>>>>>
> >>>>>>> On Tue, Jul 22, 2014 at 6:12 PM, Abraham Elmahrek <
> abe@cloudera.com>
> >>>>>>> wrote:
> >>>>>>>> Hey guys,
> >>>>>>>>
> >>>>>>>> It looks like we still have a couple of Jiras still
open. Let's
> see
> >>>>>>>> how things look tomorrow, but let's have these be the
last two
> Jiras
> >>>>>>>> slotted for this release.
> >>>>>>>>
> >>>>>>>> -Abe
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Fri, Jul 18, 2014 at 1:20 PM, Venkat Ranganathan
<
> >>>>>>>> vranganathan@hortonworks.com> wrote:
> >>>>>>>>
> >>>>>>>>> Sounds good.   I have already uploaded the patch
for SQOOP-1358.
> >>> It
> >>>>>>>>> needs to be reviewed by a committer and then committed
if no
> >>>>>>>>> further changes are needed (or more work needed
otherwise).
> >>>>>>>>>
> >>>>>>>>> Thanks for driving the release!
> >>>>>>>>>
> >>>>>>>>> Venkat
> >>>>>>>>>
> >>>>>>>>> On Fri, Jul 18, 2014 at 10:58 AM, Jarek Jarcec Cecho
> >>>>>>>>> <jarcec@apache.org>
> >>>>>>>>> wrote:
> >>>>>>>>>> Seems reasonable to me, thank you for driving
the release Abe!
> >>>>>>>>>>
> >>>>>>>>>> Jarcec
> >>>>>>>>>>
> >>>>>>>>>> On Jul 18, 2014, at 10:40 AM, Abraham Elmahrek
<
> abe@cloudera.com>
> >>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hey guys,
> >>>>>>>>>>>
> >>>>>>>>>>> I haven't seen a -1 on this suggestion.
So let's create a
> >>>>>>>>>>> release branch come Wednesday July 23rd.
> >>>>>>>>>>>
> >>>>>>>>>>> -Abe
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Jul 16, 2014 at 11:38 AM, Abraham
Elmahrek
> >>>>>>>>>>> <abe@cloudera.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Are there any objections to this branch
date?
> >>>>>>>>>>>>
> >>>>>>>>>>>> -Abe
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Mon, Jul 14, 2014 at 1:22 PM, Abraham
Elmahrek
> >>>>>>>>>>>> <abe@apache.org>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hey guys,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> It looks like there are a couple
of Jiras that are in
> progress
> >>>>>>>>>>>>> and slotted for the 1.4.5 release.
If we aim for July 23rd to
> >>>>>>>>>>>>> create a
> >>>>>>>>> release
> >>>>>>>>>>>>> branch, does that give every one
enough time to finish up
> what
> >>>>>>>>>>>>> they're working on?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> See https://issues.apache.org/jira/browse/SQOOP-1353
for
> more
> >>>>>>>>>>>>> information.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -Abe
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>>>> NOTICE: This message is intended for the use of
the individual or
> >>>>>>>>> entity to which it is addressed and may contain
information that
> is
> >>>>>>>>> confidential, privileged and exempt from disclosure
under
> >>>>>>>>> applicable law. If the reader of this message is
not the intended
> >>>>>>>>> recipient, you are hereby notified that any printing,
copying,
> >>>>>>>>> dissemination, distribution, disclosure or forwarding
of this
> >>>>>>>>> communication is strictly prohibited. If you have
received this
> >>>>>>>>> communication in error, please contact the sender
immediately and
> >>>>>>>>> delete it from your
> >>>>>>> system. Thank You.
> >>>>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>> NOTICE: This message is intended for the use of the individual
or
> >>>>>>> entity to which it is addressed and may contain information
that is
> >>>>>>> confidential, privileged and exempt from disclosure under
> applicable
> >>>>>>> law. If the reader of this message is not the intended recipient,
> you
> >>>>>>> are hereby notified that any printing, copying, dissemination,
> >>>>>>> distribution, disclosure or forwarding of this communication
is
> >>>>>>> strictly prohibited. If you have received this communication
in
> >>> error,
> >>>>>>> please contact the sender immediately and delete it from
your
> system.
> >>>>> Thank You.
> >>>>>>>
> >>>>>
> >>>>> --
> >>>>> CONFIDENTIALITY NOTICE
> >>>>> NOTICE: This message is intended for the use of the individual or
> >>> entity to
> >>>>> which it is addressed and may contain information that is
> confidential,
> >>>>> privileged and exempt from disclosure under applicable law. If the
> >>> reader
> >>>>> of this message is not the intended recipient, you are hereby
> notified
> >>> that
> >>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>> forwarding of this communication is strictly prohibited. If you
have
> >>>>> received this communication in error, please contact the sender
> >>> immediately
> >>>>> and delete it from your system. Thank You.
> >>>>>
> >>>
> >>>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message