spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Girish bhat m <girishbhat...@gmail.com>
Subject Re: How to get logging right for Spark applications in the YARN ecosystem
Date Fri, 02 Aug 2019 10:17:59 GMT
Hi Raman,

Since you are using YARN, you collect the yarn logs (which also contains
the application logs) by executing below command and move to the desired
location

yarn logs -applicationId <your_yarn_app_id>

Best
Girish

On Fri, Aug 2, 2019 at 10:17 AM Srinath C <srinath.c@gmail.com> wrote:

> Hi Raman,
>
> Probably use the rolling file appender in log4j to compress the rotated
> log file?
>
> Regards.
>
>
> On Fri, Aug 2, 2019 at 12:47 AM raman gugnani <ramangugnani.007@gmail.com>
> wrote:
>
>> HI ,
>>
>> I am looking for right solution for logging the logs produced by the
>> executors. Most of the places I have seen logging done by log4j properties,
>> but no where people I have seen any solution where logs are being
>> compressed.
>>
>> Is there anyway I can compress the logs, So that further those logs can
>> be shipped to S3.
>>
>> --
>> Raman Gugnani
>>
>

-- 
Girish bhat m

Mime
View raw message