storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ben Gould <ben.go...@inovexcorp.com>
Subject Re: questions on task, threads and workers
Date Tue, 03 Feb 2015 15:30:18 GMT
Just a guess, but if you said:

topologyBuilder.setBolt("green-bolt", new GreenBolt(), 2).setNumTasks(1) 
   (2 executor threads, but only 1 task)

Then you'd get two executors with 1 task each?


On 02/02/2015 08:15 PM, Kosala Dissanayake wrote:
> Yes it specifies the number of executors.
>
> *But by default, storm assigns one task per executor. *
>
> Therefore, when you set the number of executors, by default, this will 
> be equal to the number of tasks.
>
>
> If you wish to, you can override the one task per executor default and 
> manually set the number of tasks
>
> topologyBuilder.setBolt("green-bolt", new GreenBolt(), 2).setNumTasks(4)
>
> This will set 4 tasks to the green bolt. Since the parallelism hint is 
> 2, there will be 2 executors. Therefore, each executor will get 4/2 = 
> 2 tasks.
>
>
> Your other question is whether we can do this
>
> topologyBuilder.setBolt("green-bolt", new GreenBolt(), 
> 2).setNumTasks(1)   (2 executor threads, but only 1 task)
>
> I do not know off the top of my head but I suspect that you can't do 
> this. Maybe you can it out.
>
>
> You can read this thread as well to understand this better
> http://stackoverflow.com/questions/20371073/how-to-tune-the-parallelism-hint-in-storm
>
>
> On Tue, Feb 3, 2015 at 10:44 AM, clay teahouse <clayteahouse@gmail.com 
> <mailto:clayteahouse@gmail.com>> wrote:
>
>     According to the storm docs, as of Storm 0.8
>     the/parallelism_hint/ parameter now specifies the initial number
>     of executors (not tasks!) for that bolt.
>     I assume this means that number of tasks and the number of
>     parallelism hints have to be set separately and the number of
>     executor threads (i.e., the parallelism hint) does not set the
>     value of the number of tasks. Hence my question, what happens if
>     the number of tasks is less than the number of executors?
>
>
>     On Mon, Feb 2, 2015 at 5:37 PM, Kosala Dissanayake
>     <umaradissa@gmail.com <mailto:umaradissa@gmail.com>> wrote:
>
>         1. The default number of tasks for an executor thread is 1. So
>         if you just specify the parallelism for a bolt as 3, you will
>         have 3 executor threads. Since the default num of tasks for an
>         executor thread is 1, this means you will get 3 tasks.
>
>         You can increase the number of tasks for an executor thread
>         from the default 1 by using setNumTasks.
>
>         2. I don't think that's possible.
>
>         3. No. The number of workers is just the number of processes
>         to which executors can be allocated. If you have less workers,
>         more executors will be assigned to each worker and vice versa.
>
>
>
>         http://www.michael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/
is
>         a pretty good introduction to these concepts
>
>         On Mon, Feb 2, 2015 at 9:54 AM, clay teahouse
>         <clayteahouse@gmail.com <mailto:clayteahouse@gmail.com>> wrote:
>
>             Hi,
>             I have a few simple questions.
>             1)In storm .9.x, what is the default value for the bolt
>             num tasks? According to the docs, the parallelism hint no
>             longer sets the number of tasks, but the number of
>             executor threads.
>             2)What happens if the number of tasks is less than the
>             number of threads? Should I assume this results in idle
>             threads?
>             3)Does the number of workers multiplies the number of
>             tasks and threads?
>
>             feedback appreciated,
>             Clay
>
>
>
>

Mime
View raw message