storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Cristanis <thomascrista...@gmail.com>
Subject Re: max parallelism values
Date Mon, 27 Feb 2017 19:02:45 GMT
I do not know how these mechanisms you mentioned (SQS, S3 and database),
but Kafka is recommended to set the parallelism hit number according to the
amount of Kafka partitions <https://kafka.apache.org/documentation/>. The
number of parallelism hit there is in addition to the amount of Kafka
partitions the executors have become idle.
I have seen recommendations in hortonworks <http://hortonworks.com/blog/>
that the number of executors does not exceed the number of cores.


--
Thomas Cristanis

2017-02-27 15:05 GMT-03:00 pradeep s <sreekumar.pradeep@gmail.com>:

> Hi,
> I am running a 5 worker node storm cluster on ec2 m4.2x large machines.
> Each machine is having 8 cores . I have a spout which is consuming from
> SQS and two bolts ,one to S3 and another to a database .
> What will be the maximum parallelism value i can assign for Spout and Bolt.
> Since i have 8*5= 40 cores in total, is 40 the max parallelism i can give
> for spout and bolt .
> Also if i assign 40 parallelism hint for spout, will the bolt parallelism
> value can be the same?
> Regards
> Pradeep S
>

Mime
View raw message