spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <>
Subject Re: sparkStreaming how to work with partitions,how tp create partition
Date Thu, 27 Aug 2015 10:52:11 GMT
You can use the driver ui and click on the Jobs -> Stages to see the number
of partitions being created for that job. If you want to increase the
partitions, then you could do a .repartition too. With directStream api i
guess the # partitions in spark will be equal to the number of partitions
in the kafka topic.

Best Regards

On Sat, Aug 22, 2015 at 4:09 PM, Gaurav Agarwal <>

> 1. how to work with partition in spark streaming from kafka
> 2. how to create partition in spark streaming from kafka
> when i send the message from kafka topic having three partitions.
> Spark will listen the message when i say kafkautils.createStream or
> createDirectstSream have local[4]
> Now i want to see if spark will create partitions when it receive
> message from kafka using dstream, how and where ,prwhich method of
> spark api i have to see to find out

View raw message