storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kosala Dissanayake <umaradi...@gmail.com>
Subject Re: questions on task, threads and workers
Date Tue, 03 Feb 2015 23:55:11 GMT
@Ben

No. setNumTasks refers to the *total* number of tasks for the component.

Anyway this is not possible, which is confirmed by Nathan's answer in a
different thread given below.

*"The number of executors for a component must be <= than the number of
tasks"*  (
https://groups.google.com/d/msg/storm-user/VvXCG-TqMx0/7DfWltEkzvAJ)

On Wed, Feb 4, 2015 at 2:30 AM, Ben Gould <ben.gould@inovexcorp.com> wrote:

>  Just a guess, but if you said:
>
> topologyBuilder.setBolt("green-bolt", new GreenBolt(), 2).setNumTasks(1)
> (2 executor threads, but only 1 task)
>
> Then you'd get two executors with 1 task each?
>
>
>
> On 02/02/2015 08:15 PM, Kosala Dissanayake wrote:
>
> Yes it specifies the number of executors.
>
>  *But by default, storm assigns one task per executor. *
>
>  Therefore, when you set the number of executors, by default, this will
> be equal to the number of tasks.
>
>
>  If you wish to, you can override the one task per executor default and
> manually set the number of tasks
>
>  topologyBuilder.setBolt("green-bolt", new GreenBolt(), 2).setNumTasks(4)
>
>  This will set 4 tasks to the green bolt. Since the parallelism hint is 2,
> there will be 2 executors. Therefore, each executor will get 4/2 = 2
> tasks.
>
>
>  Your other question is whether we can do this
>
>  topologyBuilder.setBolt("green-bolt", new GreenBolt(), 2).setNumTasks(1)
>   (2 executor threads, but only 1 task)
>
>  I do not know off the top of my head but I suspect that you can't do
> this. Maybe you can it out.
>
>
>  You can read this thread as well to understand this better
>
> http://stackoverflow.com/questions/20371073/how-to-tune-the-parallelism-hint-in-storm
>
>
> On Tue, Feb 3, 2015 at 10:44 AM, clay teahouse <clayteahouse@gmail.com>
> wrote:
>
>> According to the storm docs, as of Storm 0.8 the*parallelism_hint* parameter
>> now specifies the initial number of executors (not tasks!) for that bolt.
>> I assume this means that number of tasks and the number of parallelism
>> hints have to be set separately and the number of executor threads (i.e.,
>> the parallelism hint) does not set the value of the number of tasks. Hence
>> my question, what happens if the number of tasks is less than the number of
>> executors?
>>
>>
>> On Mon, Feb 2, 2015 at 5:37 PM, Kosala Dissanayake <umaradissa@gmail.com>
>> wrote:
>>
>>> 1. The default number of tasks for an executor thread is 1. So if you
>>> just specify the parallelism for a bolt as 3, you will have 3 executor
>>> threads. Since the default num of tasks for an executor thread is 1, this
>>> means you will get 3 tasks.
>>>
>>>  You can increase the number of tasks for an executor thread from the
>>> default 1 by using setNumTasks.
>>>
>>>  2. I don't think that's possible.
>>>
>>>  3. No. The number of workers is just the number of processes to which
>>> executors can be allocated. If you have less workers, more executors will
>>> be assigned to each worker and vice versa.
>>>
>>>
>>>
>>>
>>> http://www.michael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/
is
>>> a pretty good introduction to these concepts
>>>
>>> On Mon, Feb 2, 2015 at 9:54 AM, clay teahouse <clayteahouse@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>> I have a few simple questions.
>>>> 1)In storm .9.x, what is the default value for the bolt num tasks?
>>>> According to the docs, the parallelism hint no longer sets the number of
>>>> tasks, but the number of executor threads.
>>>> 2)What happens if the number of tasks is less than the number of
>>>> threads? Should I assume this results in idle threads?
>>>> 3)Does the number of workers multiplies the number of tasks and threads?
>>>>
>>>>  feedback appreciated,
>>>> Clay
>>>>
>>>
>>>
>>
>

Mime
View raw message