From user-return-5543-apmail-storm-user-archive=storm.apache.org@storm.apache.org Thu Dec 11 13:54:19 2014 Return-Path: X-Original-To: apmail-storm-user-archive@minotaur.apache.org Delivered-To: apmail-storm-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B0E9AC345 for ; Thu, 11 Dec 2014 13:54:19 +0000 (UTC) Received: (qmail 55162 invoked by uid 500); 11 Dec 2014 13:54:18 -0000 Delivered-To: apmail-storm-user-archive@storm.apache.org Received: (qmail 55113 invoked by uid 500); 11 Dec 2014 13:54:18 -0000 Mailing-List: contact user-help@storm.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@storm.apache.org Delivered-To: mailing list user@storm.apache.org Received: (qmail 55102 invoked by uid 99); 11 Dec 2014 13:54:18 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Dec 2014 13:54:18 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW X-Spam-Check-By: apache.org Received-SPF: error (nike.apache.org: local policy) Received: from [209.85.192.42] (HELO mail-qg0-f42.google.com) (209.85.192.42) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Dec 2014 13:53:50 +0000 Received: by mail-qg0-f42.google.com with SMTP id q108so1723304qgd.15 for ; Thu, 11 Dec 2014 05:52:44 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:mime-version:message-id:in-reply-to :references:from:to:cc:subject:content-type; bh=m5O9u5hEbcuayT/oZEt3crVo7HysY66E2VAGUxednwE=; b=kaC2SBX106CpUJTICHaZakIbWIQg4zA30JTEnB+JL3NPJi2xI/7rfvG7scMlM+O4+W lAkGMnJ3jDjXNH0VE5+XL8Gr4f2Kv4N3UzZC13xSDMv4TLTNMQh3qv5VOqOkmAIUn3cj yZ9lK3KRRCYlt0fVS0Bzd4/xxvKSl0tZwQlifnBIQJTJmhZRZwun+ssNVpvM61gHUNLd HUPcM3f7C1N9ec7Qsey/6e1F3trf8LJFoHJIxTokCyaS63m1DgDF6MLGYhx9+eRHulbR SyZtwBiBeOqu7gCGVzU2PUSvFMMIQFmuVmjuC/hzaqY2KL0iJsOA9zqwXksaXdi1J+J0 zjCQ== X-Gm-Message-State: ALoCoQkSsQvp3qPBh5YEId85kQGCGrHSLOUlA4Yr/DOGMAUGzVqwn/uWlrxboK0hmxREsqQjWxfz X-Received: by 10.224.14.133 with SMTP id g5mr19884694qaa.81.1418305964639; Thu, 11 Dec 2014 05:52:44 -0800 (PST) Received: from hedwig-33.prd.orcali.com (ec2-54-85-253-40.compute-1.amazonaws.com. [54.85.253.40]) by mx.google.com with ESMTPSA id d9sm1065403qam.26.2014.12.11.05.52.43 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 11 Dec 2014 05:52:43 -0800 (PST) Date: Thu, 11 Dec 2014 05:52:43 -0800 (PST) X-Google-Original-Date: Thu, 11 Dec 2014 13:52:43 GMT MIME-Version: 1.0 X-Mailer: Nodemailer (0.5.0; +http://www.nodemailer.com/) Message-Id: <1418305963495.144f2707@Nodemailer> In-Reply-To: References: X-Orchestra-Oid: C44F2F51-F539-47D3-BC86-489D8547E95E X-Orchestra-Sig: 1db8ee950c13c573dd117871b833b6464d5b6a6e X-Orchestra-Thrid: 0D45E57F-3283-4496-AB59-A7BAB3DD14D8 X-Orchestra-Thrid-Sig: 7382d33603b4d0b13406ddc0e0f63f04882ff1f5 X-Orchestra-Account: 20b421ab73d0426add1f039e73b54c5af8ea70b8 From: "Huy Le Van" To: user@storm.apache.org Cc: user@storm.apache.org Subject: Re: TransactionalTridentKafkaSpout using only 1 executor Content-Type: multipart/mixed; boundary="----Nodemailer-0.5.0-?=_1-1418305963899" X-Virus-Checked: Checked by ClamAV on apache.org ------Nodemailer-0.5.0-?=_1-1418305963899 Content-Type: multipart/alternative; boundary="----Nodemailer-0.5.0-?=_2-1418305963899" ------Nodemailer-0.5.0-?=_2-1418305963899 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Taylor, After writing my own producer with a custom round-robin partitioner, it = seems the problem is solved now. Here=E2=80=99s my topology: =C2=A0 =C2=A0 =C2=A0 =C2=A0 BrokerHosts brokerHosts =3D new = ZkHosts(=5FkafkaZkConnect); =C2=A0 =C2=A0 =C2=A0 =C2=A0 TridentKafkaConfig kafkaConfig =3D new = TridentKafkaConfig(brokerHosts, =5Ftopic); =C2=A0 =C2=A0 =C2=A0 =C2=A0 Stream inputStream =3D =5Ftopology.= newStream(=22bytes=22, new TransactionalTridentKafkaSpout(kafkaConfig)) =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .= parallelismHint(=5FnumKafkaPartitions).name(=22reading-kafka=22) =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .each(new = Fields(=22bytes=22), new Reporter(graphiteConfig, =22tuples=22, = =22received=22)).name(=22reporting-to-graphite=22); Best regards,Huy, Le Van INSIGHT@ NUI Galway Lower Dangan, IDA Business Park,Galway, Ireland On Wednesday, Dec 10, 2014 at 6:52 p.m., P. Taylor Goetz , wrote: Can you share your topology code=3F It sound like you may not be setting = the parallelism hint in the right place. -Taylor On Dec 4, 2014, at 3:32 PM, Huy Le Van = wrote: Could someone help me please=3F Best regards,Huy, Le Van On Thursday, Dec 4, 2014 at 3:35 p.m., Huy Le Van , wrote: Hi, I=E2=80=99m trying to tune Kafka Trident (Transactional) and seeing that = the =E2=80=98spout0=E2=80=99 bolt uses only one executor. The problem is = exactly as described in https://groups.google.com/forum/#!= msg/storm-user/bI7976v9R5g/fulzpnPmzkEJ However, my Kafka topic has 16 partitions and I already set parallelismHint= of TransactionalTridentKafkaSpout to 16. What am I doing wrong here=3F = Please advise. Many thanks,Huy, Le Van ------Nodemailer-0.5.0-?=_2-1418305963899 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
Hi Taylor,

After writing my own producer with a custom round-robin partitioner, = it seems the problem is solved now. Here’s my topology:

        BrokerHosts brokerHosts =3D new = ZkHosts(=5FkafkaZkConnect);
        TridentKafkaConfig kafkaConfig =3D new = TridentKafkaConfig(brokerHosts, =5Ftopic);
        Stream inputStream =3D =5Ftopology.= newStream(=22bytes=22, new TransactionalTridentKafkaSpout(kafkaConfig))
                .= parallelismHint(=5FnumKafkaPartitions).name(=22reading-kafka=22)
                .each(new = Fields(=22bytes=22), new Reporter(graphiteConfig, =22tuples=22, = =22received=22)).name(=22reporting-to-graphite=22);

Best regards,
Huy, Le Van
INSIGHT@ NUI Galway
Lower Dangan, IDA Business Park,Galway, Ireland

On Wednesday, Dec 10, 2014 at 6:52 p.m., P. Taylor Goetz <ptgoetz@gmail= .com>, wrote:
Can you share your topology code=3F It sound like you may not be = setting the parallelism hint in the right place.

-Taylor


On Dec 4, 2014, at 3:32 PM, Huy Le Van <huy.levan@insight-centre.org> = wrote:

Could someone help me please=3F

Best regards,
Huy, Le Van

On Thursday, Dec 4, 2014 at 3:35 p.m., Huy Le Van <huy.levan@insight-centre.= org>, wrote:
Hi,

I’m trying to tune Kafka Trident (Transactional) and seeing that= the ‘spout0’ bolt uses only one executor. The problem is = exactly as described in https://groups.google.= com/forum/#!msg/storm-user/bI7976v9R5g/fulzpnPmzkEJ
However, my Kafka topic has 16 partitions and I already set = parallelismHint of TransactionalTridentKafkaSpout to 16. What am I doing = wrong here=3F Please advise.

Many thanks,
Huy, Le Van

------Nodemailer-0.5.0-?=_2-1418305963899-- ------Nodemailer-0.5.0-?=_1-1418305963899 Content-Type: application/pgp-signature; name="signature.asc" Content-Disposition: attachment; filename="signature.asc" Content-Transfer-Encoding: base64 LS0tLS1CRUdJTiBQR1AgU0lHTkFUVVJFLS0tLS0NCkNvbW1lbnQ6IEdQR1Rvb2xzIC0gaHR0cHM6 Ly9ncGd0b29scy5vcmcNCg0KaVFFY0JBRUJDZ0FHQlFKVWlKWGFBQW9KRUkzZ09XTG9DNC85eWVr SC8yaC9XWHlZWEtmaGtDRC9LZ2hPMFhqcg0KN1JPS09nWTBibFdBRVc0b2xKeHZYQ3F2SDQya1ZN Z3htK0xtZTdqc051ZWppaTlYUkJkWUo4d29XRmFzVDdESg0Kb2lNMFBmaHIzdE9pbDc0T3U2MnBs SGtGNzBaVlVBUVVwdi9rRUp1UVR4N3RGMWxxaWM5S1BUOEZqVnQ1MmZYYw0KcjE5cjdYbkltdi9r RU5MbWVySHA5eHZ6V09qSU5BeTVIaS9NcWpmYTRRa2JleU50TGNNWmpvcmxtbWtEVDQrZA0KZHkz eU5keGlCL1cyclRBZ2pFQWxWeGM0TFZHZkZ1T3V5ZzViZDFzM0lNdmc1cWh3bklQbWI4WHJ3QkM5 c3pRRw0KZXNaYW9UQmlBOXF0OVNDUjhCaE5MaXlOUmRVdUx3MXV2b09iRk0zK3gyR01SNXFKVFNF SmpwWHF3YXhJQlNVPQ0KPW5BakMNCi0tLS0tRU5EIFBHUCBTSUdOQVRVUkUtLS0tLQ0K ------Nodemailer-0.5.0-?=_1-1418305963899--