spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siva <sbhavan...@gmail.com>
Subject Re: Monitoring tools for spark streaming
Date Tue, 29 Sep 2015 19:12:06 GMT
Thanks Adrian, Hari and Shixiong for your inputs. We are using Spark 1.2.1.

Explored Streaming UI, it has very good stats. Thanks for the support

- Siva

On Tue, Sep 29, 2015 at 5:33 AM, Adrian Tanase <atanase@adobe.com> wrote:

> You can also use the REST api introduced in 1.4 – although it’s harder to
> parse:
>
>    - jobs from the same batch are not grouped together
>    - You only get total delay, not scheduling delay
>
>
> From: Hari Shreedharan
> Date: Tuesday, September 29, 2015 at 5:27 AM
> To: Shixiong Zhu
> Cc: Siva, "user@spark.apache.org"
> Subject: Re: Monitoring tools for spark streaming
>
> +1. The Streaming UI should give you more than enough information.
>
> Thanks,
> Hari
>
>
> On Mon, Sep 28, 2015 at 9:55 PM, Shixiong Zhu <zsxwing@gmail.com> wrote:
>
>> Which version are you using? Could you take a look at the new Streaming
>> UI in 1.4.0?
>>
>> Best Regards,
>> Shixiong Zhu
>>
>> 2015-09-29 7:52 GMT+08:00 Siva <sbhavanari@gmail.com>:
>>
>>> Hi,
>>>
>>> Could someone recommend the monitoring tools for spark streaming?
>>>
>>> By extending StreamingListener we can dump the delay in processing of
>>> batches and some alert messages.
>>>
>>> But are there any Web UI tools where we can monitor failures, see delays
>>> in processing, error messages and setup alerts etc.
>>>
>>> Thanks
>>>
>>>
>>
>

Mime
View raw message