storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Cristanis <thomascrista...@gmail.com>
Subject Kafkaspout very slow
Date Tue, 07 Feb 2017 12:53:21 GMT
Hi,

I'm having a problem that I've tried a lot, and I can not find an answer.
I'm having slowness problem at the moment reading Kafka's data with
Trident. For example in about 10 minutes were issued in the spout less than
120 thousand messages.

BrokerHosts hosts = new ZkHosts(Configuration.ZOOKEEPER_CONNECTION_STRING);

TridentKafkaConfig spoutConf = new TridentKafkaConfig(hosts, topicName);

spoutConf.scheme = new SchemeAsMultiScheme(new StringScheme());

spoutConf.ignoreZkOffsets = true;

spoutConf.startOffsetTime = -1;

spoutConf.bufferSizeBytes = 1024 * 1024 * 4;

spoutConf.fetchSizeBytes = 1024 * 1024 * 4;


OpaqueTridentKafkaSpout spout = new OpaqueTridentKafkaSpout(spoutConf);

Stream stream = topologyTrident.newStream("lines",
spout).parallelismHint(8);



--
Thomas Cristanis

Mime
View raw message