storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kosala Dissanayake <umaradi...@gmail.com>
Subject Re: questions on task, threads and workers
Date Tue, 03 Feb 2015 01:15:09 GMT
Yes it specifies the number of executors.

*But by default, storm assigns one task per executor. *

Therefore, when you set the number of executors, by default, this will be
equal to the number of tasks.


If you wish to, you can override the one task per executor default and
manually set the number of tasks

topologyBuilder.setBolt("green-bolt", new GreenBolt(), 2).setNumTasks(4)

This will set 4 tasks to the green bolt. Since the parallelism hint is 2,
there will be 2 executors. Therefore, each executor will get 4/2 = 2 tasks.


Your other question is whether we can do this

topologyBuilder.setBolt("green-bolt", new GreenBolt(), 2).setNumTasks(1)
(2 executor threads, but only 1 task)

I do not know off the top of my head but I suspect that you can't do this.
Maybe you can it out.


You can read this thread as well to understand this better
http://stackoverflow.com/questions/20371073/how-to-tune-the-parallelism-hint-in-storm


On Tue, Feb 3, 2015 at 10:44 AM, clay teahouse <clayteahouse@gmail.com>
wrote:

> According to the storm docs, as of Storm 0.8 the*parallelism_hint* parameter
> now specifies the initial number of executors (not tasks!) for that bolt.
> I assume this means that number of tasks and the number of parallelism
> hints have to be set separately and the number of executor threads (i.e.,
> the parallelism hint) does not set the value of the number of tasks. Hence
> my question, what happens if the number of tasks is less than the number of
> executors?
>
>
> On Mon, Feb 2, 2015 at 5:37 PM, Kosala Dissanayake <umaradissa@gmail.com>
> wrote:
>
>> 1. The default number of tasks for an executor thread is 1. So if you
>> just specify the parallelism for a bolt as 3, you will have 3 executor
>> threads. Since the default num of tasks for an executor thread is 1, this
>> means you will get 3 tasks.
>>
>> You can increase the number of tasks for an executor thread from the
>> default 1 by using setNumTasks.
>>
>> 2. I don't think that's possible.
>>
>> 3. No. The number of workers is just the number of processes to which
>> executors can be allocated. If you have less workers, more executors will
>> be assigned to each worker and vice versa.
>>
>>
>>
>>
>> http://www.michael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/
is
>> a pretty good introduction to these concepts
>>
>> On Mon, Feb 2, 2015 at 9:54 AM, clay teahouse <clayteahouse@gmail.com>
>> wrote:
>>
>>> Hi,
>>> I have a few simple questions.
>>> 1)In storm .9.x, what is the default value for the bolt num tasks?
>>> According to the docs, the parallelism hint no longer sets the number of
>>> tasks, but the number of executor threads.
>>> 2)What happens if the number of tasks is less than the number of
>>> threads? Should I assume this results in idle threads?
>>> 3)Does the number of workers multiplies the number of tasks and threads?
>>>
>>> feedback appreciated,
>>> Clay
>>>
>>
>>
>

Mime
View raw message