storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Neilson <arsneil...@gmail.com>
Subject Re: TransactionalTridentKafkaSpout using only 1 executor
Date Thu, 04 Dec 2014 21:58:16 GMT
How is the kafka topic you are reading from partitioned? By default, kafka
will write to a single random partition at a time for 10 minutes before
switching to another. So if you are looking at live data, you would only
see data in one partition at a time unless you use a different partitioning
scheme.

See the Kafka FAQ for details on this
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified
?

On Thu, Dec 4, 2014 at 1:51 PM, Harsha <storm@harsha.io> wrote:

>  can you post your storm UI executors page image. If there are 16
> executors but only 1 seems to have fetching data. Can you please check on
> your kafka producer if its distributing your data among all of your
> partitions.
>
>
> On Thu, Dec 4, 2014, at 12:32 PM, Huy Le Van wrote:
>
>
> Could someone help me please?
>
> Best regards,
> Huy, Le Van
>
> On Thursday, Dec 4, 2014 at 3:35 p.m., Huy Le Van <
> huy.levan@insight-centre.org>, wrote:
>
>
> Hi,
>
> I’m trying to tune Kafka Trident (Transactional) and seeing that the
> ‘spout0’ bolt uses only one executor. The problem is exactly as described
> in
> https://groups.google.com/forum/#!msg/storm-user/bI7976v9R5g/fulzpnPmzkEJ
> However, my Kafka topic has 16 partitions and I already set
> parallelismHint of TransactionalTridentKafkaSpout to 16. What am I doing
> wrong here? Please advise.
>
> Many thanks,
> Huy, Le Van
>
>
>
>
>
>

Mime
View raw message