spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zahid Rahman <zahidr1...@gmail.com>
Subject Re: Can you view thread dumps on spark UI if job finished
Date Thu, 09 Apr 2020 00:24:42 GMT
http://spark.apache.org/docs/latest/monitoring.html#spark-configuration-options

says
"Note that this information is only available for the duration of the
application by default. To view the web UI after the fact, set
spark.eventLog.enabled to true before starting the application. This
configures Spark to log Spark events that encode the information displayed
in the UI to persisted storage."

Backbutton.co.uk
¯\_(ツ)_/¯
♡۶Java♡۶RMI ♡۶
Make Use Method {MUM}
makeuse.org
<http://www.backbutton.co.uk>


On Thu, 9 Apr 2020 at 00:50, Ruijing Li <liruijing09@gmail.com> wrote:

> Thanks Zahid, Yes I am using history server to see previous UIs.
>
>  However, my question still remains on viewing old thread dumps, as I
> cannot see them on the old completed spark UIs, only when spark context is
> running.
>
> On Wed, Apr 8, 2020 at 4:01 PM Zahid Rahman <zahidr1000@gmail.com> wrote:
>
>> Spark UI is only available while SparkContext is running.
>>
>> However  You can get to the Spark UI after your application  completes or
>> crashes.
>>
>> To do this Spark includes a tool called the Spark History Server that
>> allows you to reconstruct the Spark UI.
>>
>> You can find up to date information on how to use this tool in the spark
>> documentation https://spark.apache.org/docs/latest/monitoring.html
>>
>>
>>
>>
>>
>> On Wed, 8 Apr 2020, 23:47 Ruijing Li, <liruijing09@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> As stated in title, currently when I view the spark UI of a completed
>>> spark job, I see there are thread dump links in the executor tab, but
>>> clicking on them does nothing. Is it possible to see the thread dumps
>>> somehow even if the job finishes? On spark 2.4.5.
>>>
>>> Thanks.
>>> --
>>> Cheers,
>>> Ruijing Li
>>>
>> --
> Cheers,
> Ruijing Li
>

Mime
View raw message