tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prabhu Joseph <prabhujose.ga...@gmail.com>
Subject Re: Tez + YARN ATSv2
Date Wed, 23 Dec 2020 08:20:17 GMT
Thanks, that's very useful.

On Wed, Dec 23, 2020, 12:46 PM Jonathan Eagles <jeagles@gmail.com> wrote:

> This feature was in progress for some time but progress was halted.
>
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/TEZ-3820
>
> Many of the stakeholders can be found on that ticket
>
> The issue regarding scale depends on usage. I run atsv1.5 and write
> 300,000 history files a day. But they are written directly to hdfs and not
> the timeline server REST api. The rest app cannot handle the scale or
> bandwidth after around 8000 jobs per day IIRC. Reading from the timeline
> server has similar issues. We serve the tez UI, but bulk history files are
> parsed into a hive dB for serious queries.
>
> On Tue, Dec 22, 2020, 11:33 PM Prabhu Joseph <prabhujose.gates@gmail.com>
> wrote:
>
>> Hi All,
>>
>>   I am trying to integrate TEZ with ATSv2 as we are hitting scalability
>> issues with ATS1.5. Looks we need a HistoryLoggingService implementation
>> to
>> emit Tez HistoryEvents from Tez jobs to ATSv2 and change in Tez UI code to
>> use ATS2 REST API to fetch timeline events.
>>
>> Want to check if someone has already done this, if so can you share the
>> same.
>>
>> Thanks,
>> Prabhu Joseph
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message