storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Vogiatzis <michaelvogiat...@gmail.com>
Subject Re: Approach to parallelism
Date Sat, 03 Oct 2015 17:00:10 GMT
I will agree with Javier, one JVM per node should eliminate the number of
messages that need to be serialized.

For tuning Storm topologies you may find the following links useful:

https://gist.github.com/mrflip/5958028
https://wassermelonemann.wordpress.com/2014/01/22/tuning-storm-topologies/
Talk:
http://demo.ooyala.com/player.html?width=640&height=360&embedCode=Q1eXg5NzpKqUUzBm5WTIb6bXuiWHrRMi&videoPcode=9waHc6zKpbJKt9byfS7l4O4sn7Qn

Cheers,
Michael
@mvogiatzis <https://twitter.com/mvogiatzis>


On Sat, 3 Oct 2015 at 14:04 Javier Gonzalez <jagonzal@gmail.com> wrote:

> I would suggest sticking with a single worker per machine. It makes memory
> allocation easier and it makes inter-component communication much more
> efficient. Configure the executors with your parallelism hints to take
> advantage of all your availabe CPU cores.
>
> Regards,
> JG
>
> On Sat, Oct 3, 2015 at 12:10 AM, Kashyap Mhaisekar <kashyap.m@gmail.com>
> wrote:
>
>> Hi,
>> I was trying to come up with an approach to evaluate the parallelism
>> needed for a topology.
>>
>> Assuming I have 5 machines with 8 cores and 32 gb. And my topology has
>> one spout and 5 bolts.
>>
>> 1. Define one worker port per CPU to start off. (= 8 workers per machine
>> ie 40 workers over all)
>> 2. Each worker spawns one executor per component per worker, it
>> translates to 6 executors per worker which is 40x6= 240 executors.
>> 3. Of this, if the bolt logic is CPU intensive, then leave parallelism
>> hint  at 40 (total workers), else increase parallelism hint beyond 40 till
>> you hit a number beyond which there is no more visible performance.
>>
>> Does this look right?
>>
>> Thanks
>> Kashyap
>>
>
>
>
> --
> Javier González Nicolini
>
-- 
Michael Vogiatzis
Twitter: @mvogiatzis
http://micvog.com/

Mime
View raw message