spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <tathagata.das1...@gmail.com>
Subject Re: Error reporting/collecting for users
Date Thu, 29 Jan 2015 02:35:42 GMT
You could use foreachRDD to do the operations and then inside the
foreach create an accumulator to gather all the errors together

dstream.foreachRDD { rdd =>

   val accumulator = new Accumulator[....]

   rdd.map { ..... }.count   // whatever operation that is error prone

   // gather all errors from  accumulator and show on webpage, push to
database?
}


On Tue, Jan 27, 2015 at 9:34 PM, Tobias Pfeiffer <tgp@preferred.jp> wrote:
> Hi,
>
> On Wed, Jan 28, 2015 at 1:45 PM, Soumitra Kumar <kumar.soumitra@gmail.com>
> wrote:
>>
>> It is a Streaming application, so how/when do you plan to access the
>> accumulator on driver?
>
>
> Well... maybe there would be some user command or web interface showing the
> errors that have happened during processing...?
>
> Thanks
> Tobias
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message