sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Jarcec Cecho <jar...@apache.org>
Subject Re: Final Jiras and Release Date
Date Wed, 23 Jul 2014 21:00:56 GMT
Too late, I’m already in process of branching :-) But that’s fine - we can always cherry-pick
whatever will be necessary.

Jarcec

On Jul 23, 2014, at 1:57 PM, Abraham Elmahrek <abe@cloudera.com> wrote:

> Ah actually, it seems there is a broken OraOop test case. Let me see what's
> going on there before we branch. Jarcec, will you be free around 4PM to
> branch?
> 
> -Abe
> 
> 
> On Wed, Jul 23, 2014 at 1:19 PM, Abraham Elmahrek <abe@cloudera.com> wrote:
> 
>> Awesome! Thanks for your hard work guys!
>> 
>> Jarcec, can we use
>> https://github.com/apache/sqoop/commit/81624ddf3c8ca5834ab015ebafc8b8649ac36ab7?
>> In terms of when... let's shoot for 2PM PST?
>> 
>> -Abe
>> 
>> 
>> On Wed, Jul 23, 2014 at 12:32 PM, Jarek Jarcec Cecho <jarcec@apache.org>
>> wrote:
>> 
>>> I think that both are in. Do let me know when and from which commit you
>>> want me to branch Abe :-)
>>> 
>>> Jarcec
>>> 
>>> On Jul 23, 2014, at 10:38 AM, Abraham Elmahrek <abe@cloudera.com> wrote:
>>> 
>>>> Hey guys,
>>>> 
>>>> As soon as the Ivy and Avro version changes are in, let's start the
>>>> branching process?
>>>> 
>>>> -Abe
>>>> 
>>>> 
>>>> On Wed, Jul 23, 2014 at 1:02 AM, Venkat Ranganathan <
>>>> vranganathan@hortonworks.com> wrote:
>>>> 
>>>>> Generally  the number of tasks is a hint to InputFormat.getSplits()
>>>>> based on number of splits wanted and it can potentially return
>>>>> different number of splits.
>>>>> 
>>>>> If you see  ExportInputFormat, the split size is determined by
>>>>> combined file sizes divided by requested num mappers.   This is a
>>>>> integer division and  there is a potential for getting additional
>>>>> splits.
>>>>> 
>>>>> Thanks
>>>>> 
>>>>> Venkat
>>>>> 
>>>>> On Tue, Jul 22, 2014 at 11:41 PM, David Robson
>>>>> <David.Robson@software.dell.com> wrote:
>>>>>> Yes OraOop is not enabled by default - and also this particular issue
>>> is
>>>>> only when using partitioning - so they would have turned on some extra
>>>>> flags. Then they also have to be using 1 more mapper than requested as
>>> you
>>>>> said - so it's a pretty specific combination of events - so I don't
>>> think
>>>>> we need to hold up the release for it.
>>>>>> 
>>>>>> Do you recall what the behaviour should be in regards to this - the
>>>>> documentation leads me to believe if we request 4 mappers we should
>>> get 4
>>>>> mappers? If this is only a request and we may get more - perhaps we
>>> should
>>>>> update the documentation about this. Otherwise we could change the
>>> code to
>>>>> guarantee no more than 4 mappers?
>>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: Abraham Elmahrek [mailto:abe@cloudera.com]
>>>>>> Sent: Wednesday, 23 July 2014 3:16 PM
>>>>>> To: dev@sqoop.apache.org
>>>>>> Subject: Re: Final Jiras and Release Date
>>>>>> 
>>>>>> Hey guys,
>>>>>> 
>>>>>> AFAIK we can fall back onto the original connector? With that being
>>>>> said, OraOop is an awesome connector... So i'm completely open to
>>> waiting
>>>>> for a fix if it isn't too large.
>>>>>> 
>>>>>> David, I'm assuming you're referring to the "-m" option? I do recall
>>>>> some cases where there may be one more split than desired if splitting
>>> is
>>>>> not clean. Hopefully Venkat has more insight!
>>>>>> 
>>>>>> -Abe
>>>>>> 
>>>>>> 
>>>>>> On Tue, Jul 22, 2014 at 9:42 PM, David Robson <
>>>>> David.Robson@software.dell.com> wrote:
>>>>>> 
>>>>>>> Hey Venkat,
>>>>>>> 
>>>>>>> I had a look into that issue - it seems the problem is even though
>>> you
>>>>>>> request 4 mappers - Sqoop runs 5 mappers. Is the number of mappers
>>>>>>> meant to be guaranteed? For example if I say 4 mappers, should
I get
>>> 4
>>>>>>> mappers? I guess there could be potential to get less mappers
if the
>>>>>>> data was for example 1 row - then that would only be processed
by 1
>>>>>>> mapper. But should it be possible to get more mappers than requested?
>>>>>>> 
>>>>>>> OraOop is assuming there will be no more mappers than requested
- so
>>>>>>> if this is a valid scenario we would need to modify OraOop. On
the
>>>>>>> other hand if this is unexpected behaviour then we should look
at why
>>>>>>> the num mappers is not working?
>>>>>>> 
>>>>>>> Either way I don't think we need to fix it for this release -
as it
>>>>>>> seems customers would be unlikely to hit this issue.
>>>>>>> 
>>>>>>> David
>>>>>>> 
>>>>>>> -----Original Message-----
>>>>>>> From: Venkat Ranganathan [mailto:vranganathan@hortonworks.com]
>>>>>>> Sent: Wednesday, 23 July 2014 1:04 PM
>>>>>>> To: dev@sqoop.apache.org
>>>>>>> Subject: Re: Final Jiras and Release Date
>>>>>>> 
>>>>>>> Abe
>>>>>>> 
>>>>>>> Do we want to target SQOOP-1388 also for 1.4.5 - the one that
Vidya
>>>>>>> has raised for one of the Oracle connector export failure?
>>>>>>> 
>>>>>>> My avro patch is in RB.
>>>>>>> 
>>>>>>> Thanks
>>>>>>> 
>>>>>>> Venkat
>>>>>>> 
>>>>>>> On Tue, Jul 22, 2014 at 6:12 PM, Abraham Elmahrek <abe@cloudera.com>
>>>>>>> wrote:
>>>>>>>> Hey guys,
>>>>>>>> 
>>>>>>>> It looks like we still have a couple of Jiras still open.
Let's see
>>>>>>>> how things look tomorrow, but let's have these be the last
two Jiras
>>>>>>>> slotted for this release.
>>>>>>>> 
>>>>>>>> -Abe
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Fri, Jul 18, 2014 at 1:20 PM, Venkat Ranganathan <
>>>>>>>> vranganathan@hortonworks.com> wrote:
>>>>>>>> 
>>>>>>>>> Sounds good.   I have already uploaded the patch for
SQOOP-1358.
>>> It
>>>>>>>>> needs to be reviewed by a committer and then committed
if no
>>>>>>>>> further changes are needed (or more work needed otherwise).
>>>>>>>>> 
>>>>>>>>> Thanks for driving the release!
>>>>>>>>> 
>>>>>>>>> Venkat
>>>>>>>>> 
>>>>>>>>> On Fri, Jul 18, 2014 at 10:58 AM, Jarek Jarcec Cecho
>>>>>>>>> <jarcec@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>> Seems reasonable to me, thank you for driving the
release Abe!
>>>>>>>>>> 
>>>>>>>>>> Jarcec
>>>>>>>>>> 
>>>>>>>>>> On Jul 18, 2014, at 10:40 AM, Abraham Elmahrek <abe@cloudera.com>
>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hey guys,
>>>>>>>>>>> 
>>>>>>>>>>> I haven't seen a -1 on this suggestion. So let's
create a
>>>>>>>>>>> release branch come Wednesday July 23rd.
>>>>>>>>>>> 
>>>>>>>>>>> -Abe
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Wed, Jul 16, 2014 at 11:38 AM, Abraham Elmahrek
>>>>>>>>>>> <abe@cloudera.com>
>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Are there any objections to this branch date?
>>>>>>>>>>>> 
>>>>>>>>>>>> -Abe
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Mon, Jul 14, 2014 at 1:22 PM, Abraham
Elmahrek
>>>>>>>>>>>> <abe@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hey guys,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> It looks like there are a couple of Jiras
that are in progress
>>>>>>>>>>>>> and slotted for the 1.4.5 release. If
we aim for July 23rd to
>>>>>>>>>>>>> create a
>>>>>>>>> release
>>>>>>>>>>>>> branch, does that give every one enough
time to finish up what
>>>>>>>>>>>>> they're working on?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> See https://issues.apache.org/jira/browse/SQOOP-1353
for more
>>>>>>>>>>>>> information.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> -Abe
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>> NOTICE: This message is intended for the use of the individual
or
>>>>>>>>> entity to which it is addressed and may contain information
that is
>>>>>>>>> confidential, privileged and exempt from disclosure under
>>>>>>>>> applicable law. If the reader of this message is not
the intended
>>>>>>>>> recipient, you are hereby notified that any printing,
copying,
>>>>>>>>> dissemination, distribution, disclosure or forwarding
of this
>>>>>>>>> communication is strictly prohibited. If you have received
this
>>>>>>>>> communication in error, please contact the sender immediately
and
>>>>>>>>> delete it from your
>>>>>>> system. Thank You.
>>>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual
or
>>>>>>> entity to which it is addressed and may contain information that
is
>>>>>>> confidential, privileged and exempt from disclosure under applicable
>>>>>>> law. If the reader of this message is not the intended recipient,
you
>>>>>>> are hereby notified that any printing, copying, dissemination,
>>>>>>> distribution, disclosure or forwarding of this communication
is
>>>>>>> strictly prohibited. If you have received this communication
in
>>> error,
>>>>>>> please contact the sender immediately and delete it from your
system.
>>>>> Thank You.
>>>>>>> 
>>>>> 
>>>>> --
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>> entity to
>>>>> which it is addressed and may contain information that is confidential,
>>>>> privileged and exempt from disclosure under applicable law. If the
>>> reader
>>>>> of this message is not the intended recipient, you are hereby notified
>>> that
>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>> received this communication in error, please contact the sender
>>> immediately
>>>>> and delete it from your system. Thank You.
>>>>> 
>>> 
>>> 
>> 


Mime
View raw message