storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Xor <andreas.gramme...@gmail.com>
Subject Re: Kafkaspout very slow
Date Tue, 07 Feb 2017 12:58:01 GMT
Hello Thomas,

 Can you give us a bit more information about how things are being run?
Does this happen on remote clusters? Local? Single machine? Kafka is run on
the same cluster? There are a lot of things that might affect spout
performance in general that unfortunately we cannot discern by a few lines
of code... Please give us information on the architecture you are using, so
we can help you.

Warm regards,

A.

On Tue, Feb 7, 2017 at 12:53 PM, Thomas Cristanis <thomascristanis@gmail.com
> wrote:

> Hi,
>
> I'm having a problem that I've tried a lot, and I can not find an answer.
> I'm having slowness problem at the moment reading Kafka's data with
> Trident. For example in about 10 minutes were issued in the spout less than
> 120 thousand messages.
>
> BrokerHosts hosts = new ZkHosts(Configuration.ZOOKEEPER_CONNECTION_STRING
> );
>
> TridentKafkaConfig spoutConf = new TridentKafkaConfig(hosts, topicName);
>
> spoutConf.scheme = new SchemeAsMultiScheme(new StringScheme());
>
> spoutConf.ignoreZkOffsets = true;
>
> spoutConf.startOffsetTime = -1;
>
> spoutConf.bufferSizeBytes = 1024 * 1024 * 4;
>
> spoutConf.fetchSizeBytes = 1024 * 1024 * 4;
>
>
> OpaqueTridentKafkaSpout spout = new OpaqueTridentKafkaSpout(spoutConf);
>
> Stream stream = topologyTrident.newStream("lines",
> spout).parallelismHint(8);
>
>
>
> --
> Thomas Cristanis
>

Mime
View raw message