storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nathan Leung <ncle...@gmail.com>
Subject Re: Basic storm question
Date Wed, 02 Apr 2014 18:05:13 GMT
No, it creates an extra executor to deal with processing the ack messages
that are sent by the bolts after processing tuples.  See the following for
details on how acking works in storm:
https://github.com/nathanmarz/storm/wiki/Guaranteeing-message-processing.
 By default storm will create 1 acker per worker you have in your topology.


On Wed, Apr 2, 2014 at 2:01 PM, Huiliang Zhang <zhlntu@gmail.com> wrote:

> Hi Nathan,
>
> The last bolt just emits the tuples and no more bolt in the topology will
> consume and ack the tuples. Do you mean that storm automatically creates an
> extra executor to deal with the tuples?
>
> Thanks,
> Huiliang
>
>
> On Wed, Apr 2, 2014 at 8:31 AM, Nathan Leung <ncleung@gmail.com> wrote:
>
>> the extra task/executor is the acker thread.
>>
>>
>> On Tue, Apr 1, 2014 at 9:23 PM, Huiliang Zhang <zhlntu@gmail.com> wrote:
>>
>>> I just submitted ExclamationTopology for testing.
>>>
>>>     builder.setSpout("word", new TestWordSpout(), 10);
>>>
>>>     builder.setBolt("exclaim1", new ExclamationBolt(),
>>> 3).shuffleGrouping("word");
>>>
>>>     builder.setBolt("exclaim2", new ExclamationBolt(),
>>> 2).shuffleGrouping("exclaim1");
>>>
>>> I am supposed to see 15 executors. However, I see 16 executors and 16
>>> tasks on topology summary on storm UI. The numbers of executors are correct
>>> for the specific spout and bolts and aggregate to 15. Is that a bug in
>>> displaying topology summary?
>>>
>>> My cluster consists of 2 supervisors and each has 4 workers defined.
>>>
>>> Thanks.
>>>
>>>
>>> On Tue, Apr 1, 2014 at 1:43 PM, Nathan Leung <ncleung@gmail.com> wrote:
>>>
>>>> By default supervisor nodes can run up to 4 workers.  This is
>>>> configurable in storm.yaml (for example see supervisor.slots.ports here:
>>>> https://github.com/nathanmarz/storm/blob/master/conf/defaults.yaml).
>>>>  Memory should be split between the workers.  It's a typical Java heap, so
>>>> anything running on that worker process shares the heap.
>>>>
>>>>
>>>> On Tue, Apr 1, 2014 at 4:10 PM, David Crossland <david@elastacloud.com>wrote:
>>>>
>>>>>  On said subject, how does memory allocation work I these cases?
>>>>> Assuming 1 worker per node would you just dump all the memory available
>>>>> into worker.childopts? I guess the memory pool would be shared between
the
>>>>> spawned threads as appropriate to their needs?
>>>>>
>>>>>  I'm assuming the equivalent options for supervisor/nimbus are fine
>>>>> left at defaults.  Given that the workers/spouts/bolts are the working
>>>>> parts of the topology these would where I should target available memory?
>>>>>
>>>>>  D
>>>>>
>>>>>   *From:* Huiliang Zhang <zhlntu@gmail.com>
>>>>> *Sent:* Tuesday, 1 April 2014 19:47
>>>>> *To:* user@storm.incubator.apache.org
>>>>>
>>>>>  Thanks. It should be good if there exist some example figures
>>>>> explaining the relationship between tasks, workers, and threads.
>>>>>
>>>>>
>>>>> On Sat, Mar 29, 2014 at 6:34 AM, Susheel Kumar Gadalay <
>>>>> skgadalay@gmail.com> wrote:
>>>>>
>>>>>> No, a single worker is dedicated to a single topology no matter how
>>>>>> many threads it spawns for different bolts/spouts.
>>>>>> A single worker cannot be shared across multiple topologies.
>>>>>>
>>>>>> On 3/29/14, Nathan Leung <ncleung@gmail.com> wrote:
>>>>>> > From what I have seen, the second topology is run with 1 worker
>>>>>> until you
>>>>>> > kill the first topology or add more worker slots to your cluster.
>>>>>> >
>>>>>> >
>>>>>> > On Sat, Mar 29, 2014 at 2:57 AM, Huiliang Zhang <zhlntu@gmail.com>
>>>>>> wrote:
>>>>>> >
>>>>>> >> Thanks. I am still not clear.
>>>>>> >>
>>>>>> >> Do you mean that in a single worker process, there will
be multiple
>>>>>> >> threads and each thread will handle part of a topology?
If so,
>>>>>> what does
>>>>>> >> the number of workers mean when submitting topology?
>>>>>> >>
>>>>>> >>
>>>>>> >> On Fri, Mar 28, 2014 at 11:18 PM, padma priya chitturi <
>>>>>> >> padmapriya30@gmail.com> wrote:
>>>>>> >>
>>>>>> >>> Hi,
>>>>>> >>>
>>>>>> >>> No, its not the case. No matter how many topologies
you submit,
>>>>>> the
>>>>>> >>> workers will be shared among the topologies.
>>>>>> >>>
>>>>>> >>> Thanks,
>>>>>> >>> Padma Ch
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> On Sat, Mar 29, 2014 at 5:11 AM, Huiliang Zhang <zhlntu@gmail.com
>>>>>> >
>>>>>> >>> wrote:
>>>>>> >>>
>>>>>> >>>> Hi,
>>>>>> >>>>
>>>>>> >>>> I have a simple question about storm.
>>>>>> >>>>
>>>>>> >>>> My cluster has just 1 supervisor and 4 ports are
defined to run 4
>>>>>> >>>> workers. I first submit a topology which needs 3
workers. Then I
>>>>>> submit
>>>>>> >>>> another topology which needs 2 workers. Does this
mean that the
>>>>>> 2nd
>>>>>> >>>> topology will never be run?
>>>>>> >>>>
>>>>>> >>>> Thanks,
>>>>>> >>>> Huiliang
>>>>>> >>>>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>
>>>>>> >
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message