sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abraham Elmahrek <...@cloudera.com>
Subject Re: Final Jiras and Release Date
Date Wed, 23 Jul 2014 20:57:02 GMT
Ah actually, it seems there is a broken OraOop test case. Let me see what's
going on there before we branch. Jarcec, will you be free around 4PM to
branch?

-Abe


On Wed, Jul 23, 2014 at 1:19 PM, Abraham Elmahrek <abe@cloudera.com> wrote:

> Awesome! Thanks for your hard work guys!
>
> Jarcec, can we use
> https://github.com/apache/sqoop/commit/81624ddf3c8ca5834ab015ebafc8b8649ac36ab7?
> In terms of when... let's shoot for 2PM PST?
>
> -Abe
>
>
> On Wed, Jul 23, 2014 at 12:32 PM, Jarek Jarcec Cecho <jarcec@apache.org>
> wrote:
>
>> I think that both are in. Do let me know when and from which commit you
>> want me to branch Abe :-)
>>
>> Jarcec
>>
>> On Jul 23, 2014, at 10:38 AM, Abraham Elmahrek <abe@cloudera.com> wrote:
>>
>> > Hey guys,
>> >
>> > As soon as the Ivy and Avro version changes are in, let's start the
>> > branching process?
>> >
>> > -Abe
>> >
>> >
>> > On Wed, Jul 23, 2014 at 1:02 AM, Venkat Ranganathan <
>> > vranganathan@hortonworks.com> wrote:
>> >
>> >> Generally  the number of tasks is a hint to InputFormat.getSplits()
>> >> based on number of splits wanted and it can potentially return
>> >> different number of splits.
>> >>
>> >> If you see  ExportInputFormat, the split size is determined by
>> >> combined file sizes divided by requested num mappers.   This is a
>> >> integer division and  there is a potential for getting additional
>> >> splits.
>> >>
>> >> Thanks
>> >>
>> >> Venkat
>> >>
>> >> On Tue, Jul 22, 2014 at 11:41 PM, David Robson
>> >> <David.Robson@software.dell.com> wrote:
>> >>> Yes OraOop is not enabled by default - and also this particular issue
>> is
>> >> only when using partitioning - so they would have turned on some extra
>> >> flags. Then they also have to be using 1 more mapper than requested as
>> you
>> >> said - so it's a pretty specific combination of events - so I don't
>> think
>> >> we need to hold up the release for it.
>> >>>
>> >>> Do you recall what the behaviour should be in regards to this - the
>> >> documentation leads me to believe if we request 4 mappers we should
>> get 4
>> >> mappers? If this is only a request and we may get more - perhaps we
>> should
>> >> update the documentation about this. Otherwise we could change the
>> code to
>> >> guarantee no more than 4 mappers?
>> >>>
>> >>> -----Original Message-----
>> >>> From: Abraham Elmahrek [mailto:abe@cloudera.com]
>> >>> Sent: Wednesday, 23 July 2014 3:16 PM
>> >>> To: dev@sqoop.apache.org
>> >>> Subject: Re: Final Jiras and Release Date
>> >>>
>> >>> Hey guys,
>> >>>
>> >>> AFAIK we can fall back onto the original connector? With that being
>> >> said, OraOop is an awesome connector... So i'm completely open to
>> waiting
>> >> for a fix if it isn't too large.
>> >>>
>> >>> David, I'm assuming you're referring to the "-m" option? I do recall
>> >> some cases where there may be one more split than desired if splitting
>> is
>> >> not clean. Hopefully Venkat has more insight!
>> >>>
>> >>> -Abe
>> >>>
>> >>>
>> >>> On Tue, Jul 22, 2014 at 9:42 PM, David Robson <
>> >> David.Robson@software.dell.com> wrote:
>> >>>
>> >>>> Hey Venkat,
>> >>>>
>> >>>> I had a look into that issue - it seems the problem is even though
>> you
>> >>>> request 4 mappers - Sqoop runs 5 mappers. Is the number of mappers
>> >>>> meant to be guaranteed? For example if I say 4 mappers, should I
get
>> 4
>> >>>> mappers? I guess there could be potential to get less mappers if
the
>> >>>> data was for example 1 row - then that would only be processed by
1
>> >>>> mapper. But should it be possible to get more mappers than requested?
>> >>>>
>> >>>> OraOop is assuming there will be no more mappers than requested
- so
>> >>>> if this is a valid scenario we would need to modify OraOop. On the
>> >>>> other hand if this is unexpected behaviour then we should look at
why
>> >>>> the num mappers is not working?
>> >>>>
>> >>>> Either way I don't think we need to fix it for this release - as
it
>> >>>> seems customers would be unlikely to hit this issue.
>> >>>>
>> >>>> David
>> >>>>
>> >>>> -----Original Message-----
>> >>>> From: Venkat Ranganathan [mailto:vranganathan@hortonworks.com]
>> >>>> Sent: Wednesday, 23 July 2014 1:04 PM
>> >>>> To: dev@sqoop.apache.org
>> >>>> Subject: Re: Final Jiras and Release Date
>> >>>>
>> >>>> Abe
>> >>>>
>> >>>> Do we want to target SQOOP-1388 also for 1.4.5 - the one that Vidya
>> >>>> has raised for one of the Oracle connector export failure?
>> >>>>
>> >>>> My avro patch is in RB.
>> >>>>
>> >>>> Thanks
>> >>>>
>> >>>> Venkat
>> >>>>
>> >>>> On Tue, Jul 22, 2014 at 6:12 PM, Abraham Elmahrek <abe@cloudera.com>
>> >>>> wrote:
>> >>>>> Hey guys,
>> >>>>>
>> >>>>> It looks like we still have a couple of Jiras still open. Let's
see
>> >>>>> how things look tomorrow, but let's have these be the last two
Jiras
>> >>>>> slotted for this release.
>> >>>>>
>> >>>>> -Abe
>> >>>>>
>> >>>>>
>> >>>>> On Fri, Jul 18, 2014 at 1:20 PM, Venkat Ranganathan <
>> >>>>> vranganathan@hortonworks.com> wrote:
>> >>>>>
>> >>>>>> Sounds good.   I have already uploaded the patch for SQOOP-1358.
>> It
>> >>>>>> needs to be reviewed by a committer and then committed if
no
>> >>>>>> further changes are needed (or more work needed otherwise).
>> >>>>>>
>> >>>>>> Thanks for driving the release!
>> >>>>>>
>> >>>>>> Venkat
>> >>>>>>
>> >>>>>> On Fri, Jul 18, 2014 at 10:58 AM, Jarek Jarcec Cecho
>> >>>>>> <jarcec@apache.org>
>> >>>>>> wrote:
>> >>>>>>> Seems reasonable to me, thank you for driving the release
Abe!
>> >>>>>>>
>> >>>>>>> Jarcec
>> >>>>>>>
>> >>>>>>> On Jul 18, 2014, at 10:40 AM, Abraham Elmahrek <abe@cloudera.com>
>> >>>> wrote:
>> >>>>>>>
>> >>>>>>>> Hey guys,
>> >>>>>>>>
>> >>>>>>>> I haven't seen a -1 on this suggestion. So let's
create a
>> >>>>>>>> release branch come Wednesday July 23rd.
>> >>>>>>>>
>> >>>>>>>> -Abe
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> On Wed, Jul 16, 2014 at 11:38 AM, Abraham Elmahrek
>> >>>>>>>> <abe@cloudera.com>
>> >>>>>> wrote:
>> >>>>>>>>
>> >>>>>>>>> Are there any objections to this branch date?
>> >>>>>>>>>
>> >>>>>>>>> -Abe
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> On Mon, Jul 14, 2014 at 1:22 PM, Abraham Elmahrek
>> >>>>>>>>> <abe@apache.org>
>> >>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> Hey guys,
>> >>>>>>>>>>
>> >>>>>>>>>> It looks like there are a couple of Jiras
that are in progress
>> >>>>>>>>>> and slotted for the 1.4.5 release. If we
aim for July 23rd to
>> >>>>>>>>>> create a
>> >>>>>> release
>> >>>>>>>>>> branch, does that give every one enough
time to finish up what
>> >>>>>>>>>> they're working on?
>> >>>>>>>>>>
>> >>>>>>>>>> See https://issues.apache.org/jira/browse/SQOOP-1353
for more
>> >>>>>>>>>> information.
>> >>>>>>>>>>
>> >>>>>>>>>> -Abe
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>> CONFIDENTIALITY NOTICE
>> >>>>>> NOTICE: This message is intended for the use of the individual
or
>> >>>>>> entity to which it is addressed and may contain information
that is
>> >>>>>> confidential, privileged and exempt from disclosure under
>> >>>>>> applicable law. If the reader of this message is not the
intended
>> >>>>>> recipient, you are hereby notified that any printing, copying,
>> >>>>>> dissemination, distribution, disclosure or forwarding of
this
>> >>>>>> communication is strictly prohibited. If you have received
this
>> >>>>>> communication in error, please contact the sender immediately
and
>> >>>>>> delete it from your
>> >>>> system. Thank You.
>> >>>>>>
>> >>>>
>> >>>> --
>> >>>> CONFIDENTIALITY NOTICE
>> >>>> NOTICE: This message is intended for the use of the individual or
>> >>>> entity to which it is addressed and may contain information that
is
>> >>>> confidential, privileged and exempt from disclosure under applicable
>> >>>> law. If the reader of this message is not the intended recipient,
you
>> >>>> are hereby notified that any printing, copying, dissemination,
>> >>>> distribution, disclosure or forwarding of this communication is
>> >>>> strictly prohibited. If you have received this communication in
>> error,
>> >>>> please contact the sender immediately and delete it from your system.
>> >> Thank You.
>> >>>>
>> >>
>> >> --
>> >> CONFIDENTIALITY NOTICE
>> >> NOTICE: This message is intended for the use of the individual or
>> entity to
>> >> which it is addressed and may contain information that is confidential,
>> >> privileged and exempt from disclosure under applicable law. If the
>> reader
>> >> of this message is not the intended recipient, you are hereby notified
>> that
>> >> any printing, copying, dissemination, distribution, disclosure or
>> >> forwarding of this communication is strictly prohibited. If you have
>> >> received this communication in error, please contact the sender
>> immediately
>> >> and delete it from your system. Thank You.
>> >>
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message