storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Huy Le Van" <huy.le...@insight-centre.org>
Subject Re: TransactionalTridentKafkaSpout using only 1 executor
Date Thu, 11 Dec 2014 13:52:43 GMT
Hi Taylor,




After writing my own producer with a custom round-robin partitioner, it seems the problem
is solved now. Here’s my topology:





        BrokerHosts brokerHosts = new ZkHosts(_kafkaZkConnect);

        TridentKafkaConfig kafkaConfig = new TridentKafkaConfig(brokerHosts, _topic);

        Stream inputStream = _topology.newStream("bytes", new TransactionalTridentKafkaSpout(kafkaConfig))

                .parallelismHint(_numKafkaPartitions).name("reading-kafka")

                .each(new Fields("bytes"), new Reporter(graphiteConfig, "tuples",
"received")).name("reporting-to-graphite");





Best regards,Huy, Le Van

INSIGHT@ NUI Galway

Lower Dangan, IDA Business Park,Galway, Ireland







On Wednesday, Dec 10, 2014 at 6:52 p.m., P. Taylor Goetz <ptgoetz@gmail.com>, wrote:

Can you share your topology code? It sound like you may not be setting the parallelism hint
in the right place.




-Taylor






On Dec 4, 2014, at 3:32 PM, Huy Le Van <huy.levan@insight-centre.org> wrote:




Could someone help me please?



Best regards,Huy, Le Van






On Thursday, Dec 4, 2014 at 3:35 p.m., Huy Le Van <huy.levan@insight-centre.org>, wrote:

Hi,




I’m trying to tune Kafka Trident (Transactional) and seeing that the ‘spout0’ bolt uses
only one executor. The problem is exactly as described in https://groups.google.com/forum/#!msg/storm-user/bI7976v9R5g/fulzpnPmzkEJ


However, my Kafka topic has 16 partitions and I already set parallelismHint of TransactionalTridentKafkaSpout
to 16. What am I doing wrong here? Please advise.



Many thanks,Huy, Le Van
Mime
View raw message