spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen Song <chen.song...@gmail.com>
Subject Re: spark streaming counter metrics
Date Wed, 02 Jul 2014 17:07:56 GMT
Thanks Mayur. I will take a look at StreamingListener.

Is there any example you have handy?


On Wed, Jul 2, 2014 at 2:32 AM, Mayur Rustagi <mayur.rustagi@gmail.com>
wrote:

> You may be able to mix StreamingListener & SparkListener to get meaningful
> information about your task. however you need to connect a lot of pieces to
> make sense of the flow..
>
> Mayur Rustagi
> Ph: +1 (760) 203 3257
> http://www.sigmoidanalytics.com
> @mayur_rustagi <https://twitter.com/mayur_rustagi>
>
>
>
> On Mon, Jun 30, 2014 at 9:58 PM, Chen Song <chen.song.82@gmail.com> wrote:
>
>> I am new to spark streaming and wondering if spark streaming tracks
>> counters (e.g., how many rows in each consumer, how many rows routed to an
>> individual reduce task, etc.) in any form so I can get an idea of how data
>> is skewed? I checked spark job page but don't seem to find any.
>>
>>
>>
>> --
>> Chen Song
>>
>>
>


-- 
Chen Song

Mime
View raw message