spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From steve_ash <>
Subject Conditionally do things different on the first minibatch vs subsequent minibatches in a dstream
Date Tue, 01 Sep 2015 19:36:26 GMT
We have some logic that we need to apply while we are processing the events
in the first minibatch only.  For the second, third, etc. minibatches we
don't need to do this special logic.  I can't just do it as a one time thing
- I need to modify a field on the events in the first minibatch.

One approach:
1- create inp = inputDStream
2- call inp.foreachRdd(rdd -> rdd.foreachPartition( isFirst = true ) )
3- do m1 = if (isFirst) setStuffOnEvent )
4- do m2 = m1.mapPartition( isFirst = false )
5- do m2.foreachRdd ( rest of my stuff)

since foreachRdd in (2) is an action that just happens once right at the
beginning? Or does this happen for every minibatch just like the
foreachRdd() in (5) does?

What is another way that we accomplish this need of setting something on the
events in the first minibatch and not others?

View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message