tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Eagles <jeag...@gmail.com>
Subject Re: Tez + YARN ATSv2
Date Wed, 23 Dec 2020 07:15:46 GMT
This feature was in progress for some time but progress was halted.

https://issues.apache.org/jira/plugins/servlet/mobile#issue/TEZ-3820

Many of the stakeholders can be found on that ticket

The issue regarding scale depends on usage. I run atsv1.5 and write 300,000
history files a day. But they are written directly to hdfs and not the
timeline server REST api. The rest app cannot handle the scale or bandwidth
after around 8000 jobs per day IIRC. Reading from the timeline server has
similar issues. We serve the tez UI, but bulk history files are parsed into
a hive dB for serious queries.

On Tue, Dec 22, 2020, 11:33 PM Prabhu Joseph <prabhujose.gates@gmail.com>
wrote:

> Hi All,
>
>   I am trying to integrate TEZ with ATSv2 as we are hitting scalability
> issues with ATS1.5. Looks we need a HistoryLoggingService implementation to
> emit Tez HistoryEvents from Tez jobs to ATSv2 and change in Tez UI code to
> use ATS2 REST API to fetch timeline events.
>
> Want to check if someone has already done this, if so can you share the
> same.
>
> Thanks,
> Prabhu Joseph
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message