spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashic Mahtab <>
Subject RE: Are these numbers abnormal for spark streaming?
Date Thu, 22 Jan 2015 12:32:05 GMT
Hi Gerard,
Thanks for the response.

The messages get desrialised from msgpack format, and one of the strings is desrialised to
json. Certain fields are checked to decide if further processing is required. If so, it goes
through a series of in mem filters to check if more processing is required. If so, only then
does the "heavy" work start. That consists of a few db queries, and potential updates to the
db + message on message queue. The majority of messages don't need processing. The messages
needing processing at peak are about three every other second. 

One possible things that might be happening is the session initialisation and prepared statement
initialisation for each partition. I can resort to some tricks, but I think I'll try increasing
batch interval to 15 seconds. I'll report back with findings.


Date: Thu, 22 Jan 2015 12:30:08 +0100
Subject: Re: Are these numbers abnormal for spark streaming?

and post the code (if possible).In a nutshell, your processing time > batch interval, 
resulting in an ever-increasing delay that will end up in a crash.
3 secs to process 14 messages looks like a lot. Curious what the job logic is.
-kr, Gerard.
On Thu, Jan 22, 2015 at 12:15 PM, Tathagata Das <> wrote:
This is not normal. Its a huge scheduling delay!! Can you tell me more about the application?-
cluser setup, number of receivers, whats the computation, etc.
On Thu, Jan 22, 2015 at 3:11 AM, Ashic Mahtab <> wrote:

Hate to do this...but...erm...bump? Would really appreciate input from others using Streaming.
Or at least some docs that would tell me if these are expected or not.

Subject: Are these numbers abnormal for spark streaming?
Date: Wed, 21 Jan 2015 11:26:31 +0000

Hi Guys,
I've got Spark Streaming set up for a low data rate system (using spark's features for analysis,
rather than high throughput). Messages are coming in throughout the day, at around 1-20 per
second (finger in the air estimate...not analysed yet).  In the spark streaming UI for the
application, I'm getting the following after 17 hours.

StreamingStarted at: Tue Jan 20 16:58:43 GMT 2015Time since start: 18 hours 24 minutes 34
secondsNetwork receivers: 2Batch interval: 2 secondsProcessed batches: 16482Waiting batches:

Statistics over last 100 processed batchesReceiver StatisticsReceiverStatusLocationRecords
in last batch[2015/01/21 11:23:18]Minimum rate[records/sec]Median rate[records/sec]Maximum
rate[records/sec]Last ErrorRmqReceiver-0ACTIVEFOOOO
124726-Batch Processing StatisticsMetricLast batchMinimum25th percentileMedian75th percentileMaximumProcessing
Time3 seconds 994 ms157 ms4 seconds 16 ms4 seconds 961 ms5 seconds 3 ms5 seconds 171 msScheduling
Delay9 hours 15 minutes 4 seconds9 hours 10 minutes 54 seconds9 hours 11 minutes 56 seconds9
hours 12 minutes 57 seconds9 hours 14 minutes 5 seconds9 hours 15 minutes 4 secondsTotal Delay9
hours 15 minutes 8 seconds9 hours 10 minutes 58 seconds9 hours 12 minutes9 hours 13 minutes
2 seconds9 hours 14 minutes 10 seconds9 hours 15 minutes 8 seconds
Are these "normal". I was wondering what the scheduling delay and total delay terms are, and
if it's normal for them to be 9 hours.

I've got a standalone spark master and 4 spark nodes. The streaming app has been given 4 cores,
and it's using 1 core per worker node. The streaming app is submitted from a 5th machine,
and that machine has nothing but the driver running. The worker nodes are running alongside
Cassandra (and reading and writing to it).

Any insights would be appreciated.


View raw message