storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Huy Le Van" <>
Subject Re: TransactionalTridentKafkaSpout using only 1 executor
Date Thu, 04 Dec 2014 23:28:27 GMT
I just dumped from text files directly to kafka producer using bin/
so I guess the keys were all null. I’ll write a producer to see. By the way, what is the
command to show the distribution of my data in kafka?

Best regards,Huy, Le Van

On Thursday, Dec 4, 2014 at 11:23 p.m., Harsha <>, wrote:

It doesnt' look like your kafka producer is distributing data across the partitions. Whats
your producer looks like . Are you sending any key with each message or using null. If you
are using null than what Andrew is saying might be the problem. I would recommend using random
UUID as a key to send messages to your partition.



On Thu, Dec 4, 2014, at 03:12 PM, Huy Le Van wrote:


Hi Harsha,

I’ve attached 2 images below. You can see that I assigned 16 executors, only one seemed
to work. The other screenshot is the partition table.


Hi Andrew,

That’s an interesting. I’m quite new to Kafka. May you take a look at the second screenshot
to see if the data was distributed evenly? Let’s say it was written to one partition at
a time (yes, this is the case where I used only one producer), would it be rebalanced afterward?





Best regards,

Huy, Le Van


On Thursday, Dec 4, 2014 at 10:00 p.m., Andrew Neilson <>, wrote:

How is the kafka topic you are reading from partitioned? By default, kafka will write to a
single random partition at a time for 10 minutes before switching to another. So if you are
looking at live data, you would only see data in one partition at a time unless you use a
different partitioning scheme.


See the Kafka FAQ for details on this



On Thu, Dec 4, 2014 at 1:51 PM, Harsha <> wrote:


can you post your storm UI executors page image. If there are 16 executors but only 1 seems
to have fetching data. Can you please check on your kafka producer if its distributing your
data among all of your partitions.



On Thu, Dec 4, 2014, at 12:32 PM, Huy Le Van wrote:


Could someone help me please?


Best regards,

Huy, Le Van


On Thursday, Dec 4, 2014 at 3:35 p.m., Huy Le Van <>, wrote:




I’m trying to tune Kafka Trident (Transactional) and seeing that the ‘spout0’ bolt uses
only one executor. The problem is exactly as described in!msg/storm-user/bI7976v9R5g/fulzpnPmzkEJ

However, my Kafka topic has 16 partitions and I already set parallelismHint of TransactionalTridentKafkaSpout
to 16. What am I doing wrong here? Please advise.


Many thanks,

Huy, Le Van






Email had 2 attachments:

  165k (image/png)

  476k (image/png)
View raw message