spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From subramgr <subramanian.gir...@gmail.com>
Subject Number of records per micro-batch in DStream vs Structured Streaming
Date Tue, 03 Jul 2018 17:55:44 GMT
Hi, 

We have 2 spark streaming job one using DStreams and the other using
Structured Streaming. I have observed that the number of records per
micro-batch (Per Trigger in case of Structured Streaming) is not the same
between the two jobs. The Structured Streaming job has higher numbers
compared to the DStream job.

Is there any documentation or blog posts on how they differ and is there a
different strategy to consume data from Kafka. I know both use Kafka Direct.

The trigger was set to 60 seconds in Structured Streaming and batch size was
60 seconds as well for the DStream job.

Thanks




--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message