sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Jarcec Cecho <jar...@apache.org>
Subject Re: Final Jiras and Release Date
Date Wed, 23 Jul 2014 19:32:50 GMT
I think that both are in. Do let me know when and from which commit you want me to branch Abe
:-)

Jarcec

On Jul 23, 2014, at 10:38 AM, Abraham Elmahrek <abe@cloudera.com> wrote:

> Hey guys,
> 
> As soon as the Ivy and Avro version changes are in, let's start the
> branching process?
> 
> -Abe
> 
> 
> On Wed, Jul 23, 2014 at 1:02 AM, Venkat Ranganathan <
> vranganathan@hortonworks.com> wrote:
> 
>> Generally  the number of tasks is a hint to InputFormat.getSplits()
>> based on number of splits wanted and it can potentially return
>> different number of splits.
>> 
>> If you see  ExportInputFormat, the split size is determined by
>> combined file sizes divided by requested num mappers.   This is a
>> integer division and  there is a potential for getting additional
>> splits.
>> 
>> Thanks
>> 
>> Venkat
>> 
>> On Tue, Jul 22, 2014 at 11:41 PM, David Robson
>> <David.Robson@software.dell.com> wrote:
>>> Yes OraOop is not enabled by default - and also this particular issue is
>> only when using partitioning - so they would have turned on some extra
>> flags. Then they also have to be using 1 more mapper than requested as you
>> said - so it's a pretty specific combination of events - so I don't think
>> we need to hold up the release for it.
>>> 
>>> Do you recall what the behaviour should be in regards to this - the
>> documentation leads me to believe if we request 4 mappers we should get 4
>> mappers? If this is only a request and we may get more - perhaps we should
>> update the documentation about this. Otherwise we could change the code to
>> guarantee no more than 4 mappers?
>>> 
>>> -----Original Message-----
>>> From: Abraham Elmahrek [mailto:abe@cloudera.com]
>>> Sent: Wednesday, 23 July 2014 3:16 PM
>>> To: dev@sqoop.apache.org
>>> Subject: Re: Final Jiras and Release Date
>>> 
>>> Hey guys,
>>> 
>>> AFAIK we can fall back onto the original connector? With that being
>> said, OraOop is an awesome connector... So i'm completely open to waiting
>> for a fix if it isn't too large.
>>> 
>>> David, I'm assuming you're referring to the "-m" option? I do recall
>> some cases where there may be one more split than desired if splitting is
>> not clean. Hopefully Venkat has more insight!
>>> 
>>> -Abe
>>> 
>>> 
>>> On Tue, Jul 22, 2014 at 9:42 PM, David Robson <
>> David.Robson@software.dell.com> wrote:
>>> 
>>>> Hey Venkat,
>>>> 
>>>> I had a look into that issue - it seems the problem is even though you
>>>> request 4 mappers - Sqoop runs 5 mappers. Is the number of mappers
>>>> meant to be guaranteed? For example if I say 4 mappers, should I get 4
>>>> mappers? I guess there could be potential to get less mappers if the
>>>> data was for example 1 row - then that would only be processed by 1
>>>> mapper. But should it be possible to get more mappers than requested?
>>>> 
>>>> OraOop is assuming there will be no more mappers than requested - so
>>>> if this is a valid scenario we would need to modify OraOop. On the
>>>> other hand if this is unexpected behaviour then we should look at why
>>>> the num mappers is not working?
>>>> 
>>>> Either way I don't think we need to fix it for this release - as it
>>>> seems customers would be unlikely to hit this issue.
>>>> 
>>>> David
>>>> 
>>>> -----Original Message-----
>>>> From: Venkat Ranganathan [mailto:vranganathan@hortonworks.com]
>>>> Sent: Wednesday, 23 July 2014 1:04 PM
>>>> To: dev@sqoop.apache.org
>>>> Subject: Re: Final Jiras and Release Date
>>>> 
>>>> Abe
>>>> 
>>>> Do we want to target SQOOP-1388 also for 1.4.5 - the one that Vidya
>>>> has raised for one of the Oracle connector export failure?
>>>> 
>>>> My avro patch is in RB.
>>>> 
>>>> Thanks
>>>> 
>>>> Venkat
>>>> 
>>>> On Tue, Jul 22, 2014 at 6:12 PM, Abraham Elmahrek <abe@cloudera.com>
>>>> wrote:
>>>>> Hey guys,
>>>>> 
>>>>> It looks like we still have a couple of Jiras still open. Let's see
>>>>> how things look tomorrow, but let's have these be the last two Jiras
>>>>> slotted for this release.
>>>>> 
>>>>> -Abe
>>>>> 
>>>>> 
>>>>> On Fri, Jul 18, 2014 at 1:20 PM, Venkat Ranganathan <
>>>>> vranganathan@hortonworks.com> wrote:
>>>>> 
>>>>>> Sounds good.   I have already uploaded the patch for SQOOP-1358.
  It
>>>>>> needs to be reviewed by a committer and then committed if no
>>>>>> further changes are needed (or more work needed otherwise).
>>>>>> 
>>>>>> Thanks for driving the release!
>>>>>> 
>>>>>> Venkat
>>>>>> 
>>>>>> On Fri, Jul 18, 2014 at 10:58 AM, Jarek Jarcec Cecho
>>>>>> <jarcec@apache.org>
>>>>>> wrote:
>>>>>>> Seems reasonable to me, thank you for driving the release Abe!
>>>>>>> 
>>>>>>> Jarcec
>>>>>>> 
>>>>>>> On Jul 18, 2014, at 10:40 AM, Abraham Elmahrek <abe@cloudera.com>
>>>> wrote:
>>>>>>> 
>>>>>>>> Hey guys,
>>>>>>>> 
>>>>>>>> I haven't seen a -1 on this suggestion. So let's create a
>>>>>>>> release branch come Wednesday July 23rd.
>>>>>>>> 
>>>>>>>> -Abe
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Wed, Jul 16, 2014 at 11:38 AM, Abraham Elmahrek
>>>>>>>> <abe@cloudera.com>
>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Are there any objections to this branch date?
>>>>>>>>> 
>>>>>>>>> -Abe
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Mon, Jul 14, 2014 at 1:22 PM, Abraham Elmahrek
>>>>>>>>> <abe@apache.org>
>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hey guys,
>>>>>>>>>> 
>>>>>>>>>> It looks like there are a couple of Jiras that are
in progress
>>>>>>>>>> and slotted for the 1.4.5 release. If we aim for
July 23rd to
>>>>>>>>>> create a
>>>>>> release
>>>>>>>>>> branch, does that give every one enough time to finish
up what
>>>>>>>>>> they're working on?
>>>>>>>>>> 
>>>>>>>>>> See https://issues.apache.org/jira/browse/SQOOP-1353
for more
>>>>>>>>>> information.
>>>>>>>>>> 
>>>>>>>>>> -Abe
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> CONFIDENTIALITY NOTICE
>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>> entity to which it is addressed and may contain information that
is
>>>>>> confidential, privileged and exempt from disclosure under
>>>>>> applicable law. If the reader of this message is not the intended
>>>>>> recipient, you are hereby notified that any printing, copying,
>>>>>> dissemination, distribution, disclosure or forwarding of this
>>>>>> communication is strictly prohibited. If you have received this
>>>>>> communication in error, please contact the sender immediately and
>>>>>> delete it from your
>>>> system. Thank You.
>>>>>> 
>>>> 
>>>> --
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable
>>>> law. If the reader of this message is not the intended recipient, you
>>>> are hereby notified that any printing, copying, dissemination,
>>>> distribution, disclosure or forwarding of this communication is
>>>> strictly prohibited. If you have received this communication in error,
>>>> please contact the sender immediately and delete it from your system.
>> Thank You.
>>>> 
>> 
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>> 


Mime
View raw message