tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hitesh Shah <hit...@apache.org>
Subject Re: [DISCUSS] What versions of hadoop should we support?
Date Fri, 12 Dec 2014 00:23:24 GMT
Thanks for the useful information on your usage scenarios, folks. 

As part of TEZ-1696, to get a fully-secure UI in place, I will go ahead and update the default
dependency to hadoop 2.6 ( given that earlier versions currently do not have full secure support
). I will see if we can initiate a discussion in the Hadoop community to see if some of the
missing Timeline functionality can be back ported to the hadoop 2.4.x line ( though this will
mean that folks on hadoop 2.4.0 will need to be upgrade in any case ).

As for hadoop 2.2.x to 2.5.x, for the most part, all of Tez will continue to work as is with
some missing features depending on what features that particular version of hadoop supports.
The UI should also be functional against hadoop-2.4 for the most part except where security
is enabled. 

Details on how to build against a particular version of hadoop will be covered in the BUILDING.txt.
For details on Tez and working with various hadoop versions for YARN Timeline, details are
at: http://tez.apache.org/tez_yarn_timeline.html

And yes, please raise any issues that you see with the versions of hadoop you are running
after TEZ-1696 goes in ( sometime in the next hour or so ).

thanks
— Hitesh

On Dec 11, 2014, at 8:02 AM, Jonathan Eagles <jeagles@gmail.com> wrote:

> Adding to the discussion. I am using some 2.5 and 2.6
> 
> On Wed, Dec 10, 2014 at 8:02 PM, Cheolsoo Park <piaozhexiu@gmail.com> wrote:
>> We're using EMR 2.4 in production, and it will take a while to upgrade to
>> 2.6 depending on how fast EMR can provide us 2.6 AMI. But if that's what we
>> need to fully use Tez-8, we will upgrade our cluster.
>> 
>> On Wed, Dec 10, 2014 at 1:44 PM, Siddharth Seth <sseth@apache.org> wrote:
>>> 
>>> +1 for changing the default to 2.6.0.
>>> 
>>> Publishing multiple jars - do you mean per hadoop version supported ?
>>> That'll likely be very difficult to use - what would the dependency be ?
>>> Will there be binary compatibility issues if someone builds against tez jars
>>> built against Hadoop-2.6, and then deployed against 2.2 ?
>>> 
>>> On Mon, Dec 8, 2014 at 10:02 AM, Hitesh Shah <hitesh@apache.org> wrote:
>>>> 
>>>> Thanks for the info, Douglas. We will not be dropping support for 2.4 for
>>>> sure at the moment.
>>>> 
>>>> Anyone else have any comments in this regard?
>>>> 
>>>> @Bikas/@Sid - what do we do about publishing jars for a release? We would
>>>> need to publish 2 or 3 sets?
>>>> 
>>>> thanks
>>>> — Hitesh
>>>> 
>>>> On Dec 4, 2014, at 12:51 PM, Moore, Douglas
>>>> <Douglas.Moore@thinkbiganalytics.com> wrote:
>>>> 
>>>>> 2.4.x and beyond for the customers we have
>>>>> 
>>>>> Sent from my iPhone
>>>>> 
>>>>>> On Dec 4, 2014, at 12:36 PM, Hitesh Shah <hitesh@apache.org>
wrote:
>>>>>> 
>>>>>> Hello folks
>>>>>> 
>>>>>> Could folks who are following these mailing lists do a raise of hands
>>>>>> on which versions of Hadoop you are trying to run Tez on?
>>>>>> 
>>>>>> 1) 2.2.x
>>>>>> 2) 2.3.x
>>>>>> 3) 2.4.x
>>>>>> 4) 2.5.x
>>>>>> 5) 2.6.x
>>>>>> 
>>>>>> Given that we are building out the UI and for the large part, the
>>>>>> History UI ( i.e. the one which will display DAG details after an
app has
>>>>>> disappeared ) will be dependent on YARN Timeline, the UI will not
be
>>>>>> supported for Hadoop versions 2.2.x and 2.3.x given that they do
not support
>>>>>> Timeline.
>>>>>> Likewise, ACLs support in Timeline was introduced in Hadoop 2.6.x
-
>>>>>> this means ACLs will not be enforceable on Hadoop 2.4.x and 2.5.x
>>>>>> 
>>>>>> Also, as part of TEZ-1696, I also propose changing the default version
>>>>>> to 2.6.0 given that secure Timeline is only available as of 2.6.0.
>>>>>> 
>>>>>> Thoughts/concerns?
>>>>>> 
>>>>>> thanks
>>>>>> — Hitesh
>>>>>> 
>>>> 
>>> 
>> 


Mime
View raw message