spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Helena Edelson <>
Subject Re: Spark streaming alerting
Date Tue, 24 Mar 2015 13:39:33 GMT
Streaming _from_ cassandra, CassandraInputDStream, is coming BTW
I am working on it now.


> On Mar 23, 2015, at 5:22 AM, Khanderao Kand Gmail <> wrote:
> Akhil 
> You are right in tour answer to what Mohit wrote. However what Mohit seems to be alluring
but did not write properly might be different.
> Mohit
> You are wrong in saying "generally" streaming works in HDFS and cassandra . Streaming
typically works with streaming or queing source like Kafka, kinesis, Twitter, flume, zeroMQ,
etc (but can also from HDFS and S3 ) However , streaming context ( "receiver" wishing the
streaming context ) gets events/messages/records and forms a time window based batch (RDD)-

> So there is a maximum gap of window time from alert message was available to spark and
when the processing happens. I think you meant about this. 
> As per spark programming model, RDD is the right way to deal with data.  If you are fine
with the minimum delay of say a sec (based on min time window that dstreaming can support)
then what Rohit gave is a right model. 
> Khanderao
> On Mar 22, 2015, at 11:39 PM, Akhil Das < <>>
>> What do you mean you can't send it directly from spark workers? Here's a simple approach
which you could do:
>>     val data = ssc.textFileStream("sigmoid/")
>>     val dist = data.filter(_.contains("ERROR")).foreachRDD(rdd => alert("Errors
:" + rdd.count()))
>> And the alert() function could be anything triggering an email or sending an SMS
>> Thanks
>> Best Regards
>> On Sun, Mar 22, 2015 at 1:52 AM, Mohit Anchlia < <>>
>> Is there a module in spark streaming that lets you listen to the alerts/conditions
as they happen in the streaming module? Generally spark streaming components will execute
on large set of clusters like hdfs or Cassandra, however when it comes to alerting you generally
can't send it directly from the spark workers, which means you need a way to listen to the

View raw message