spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amit Sela <>
Subject Does MapWithState follow with a shuffle ?
Date Tue, 29 Nov 2016 21:16:52 GMT
Hi all,

I've been digging into MapWithState code (branch 1.6), and I came across
the compute
implementation in *InternalMapWithStateDStream*.

Looking at the defined partitioner
looks like it could be different from the parent RDD partitioner (if
defaultParallelism() changed for instance, or input partitioning was
smaller to begin with), which will eventually create
a ShuffleRDD.

Am I reading this right ?


View raw message