From user-return-5494-apmail-storm-user-archive=storm.apache.org@storm.apache.org Sat Dec 6 05:59:27 2014 Return-Path: X-Original-To: apmail-storm-user-archive@minotaur.apache.org Delivered-To: apmail-storm-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DC20510002 for ; Sat, 6 Dec 2014 05:59:27 +0000 (UTC) Received: (qmail 58548 invoked by uid 500); 6 Dec 2014 05:59:27 -0000 Delivered-To: apmail-storm-user-archive@storm.apache.org Received: (qmail 58505 invoked by uid 500); 6 Dec 2014 05:59:27 -0000 Mailing-List: contact user-help@storm.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@storm.apache.org Delivered-To: mailing list user@storm.apache.org Received: (qmail 58495 invoked by uid 99); 6 Dec 2014 05:59:26 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 06 Dec 2014 05:59:26 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of skgadalay@gmail.com designates 74.125.82.43 as permitted sender) Received: from [74.125.82.43] (HELO mail-wg0-f43.google.com) (74.125.82.43) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 06 Dec 2014 05:59:01 +0000 Received: by mail-wg0-f43.google.com with SMTP id l18so2461375wgh.30 for ; Fri, 05 Dec 2014 21:59:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=xim7b0lW2u7pgPV7xC0QBTbkpwGxT+mFOOGmTup0qt8=; b=BCUenajV9tUsmHjLSjPBjv+piqNJcpCjgHDa3J8BZIFuhtXlZPFr056Q4XdXwURpx+ 2Rt0mM7UIpYSbpSt3mDubIMdvRqZtH+N7Lu95+/1WFdolyy2n41S4qFjotLLeof1Oiif ewg0qBFuPDiInfvAxE9Ehmwav4eQpDJAIr1PP9HydRrvraoW6+NoVnaet111eUz1ybdM 3OsPkXhJ2mtUtSz7Hlpazr7yIDrkHNrWjXRn+aXUyvJrtOPDtalelBC0KNdXEtCwaPp+ QKKDkz8kr5c74tpOI2FeWsIulSZupHMhyVbWAXh95kKPXyIh+44ukqH5NHihCus5amVn N6VQ== MIME-Version: 1.0 X-Received: by 10.180.104.197 with SMTP id gg5mr9284373wib.7.1417845541020; Fri, 05 Dec 2014 21:59:01 -0800 (PST) Received: by 10.217.191.6 with HTTP; Fri, 5 Dec 2014 21:59:00 -0800 (PST) In-Reply-To: References: Date: Sat, 6 Dec 2014 11:29:00 +0530 Message-ID: Subject: Re: storm.yaml config: change on nimbus or worker machines? From: Susheel Kumar Gadalay To: user@storm.apache.org Content-Type: text/plain; charset=UTF-8 X-Virus-Checked: Checked by ClamAV on apache.org Workers or supervisor machines can have minimum details in storm.yaml. Nimbus server can have all the configuration details or they can be defaulted. If you want to overwrite a particular configuration property that Nimbus gives as part of topology execution down to all workers, then it can be overwritten in the topology main class before submit topology call. On 12/6/14, Erik Weathers wrote: > Thanks Susheel. Do you know enough about how storm works internally to be > able to answer my other questions? Specifically, I'd like to know how to > change the configuration back in the future; would I need to have the > topologies resubmitted in order to have the configuration passed along? Or > could I just kill all the workers and when storm creates new workers it > would propagate the changed settings from the nimbus? > > Also, which configuration options are ok to be in the worker host's > storm.yaml (since from my experiment it seemed insufficient to set this " > storm.messaging.transport" value in the worker's storm.yaml -- it *seemed* > this way because the nimbus UI was showing a different value than what I > configured in the worker storm.yaml)? Where is the distinction drawn > between a config option that needs to be in the nimbus storm.yaml versus > the supervisor/worker storm.yaml (a pointer to the code would be awesome)? > I'm still confused about how configurations are handled in storm and I'm > worried that there could be different configurations in place on workers > versus what the nimbus shows. > > - Erik > > On Fri, Dec 5, 2014 at 12:32 AM, Susheel Kumar Gadalay > > wrote: > >> If required globally for all topologies then set it in Nimbus. >> >> It will be carried to all workers when topology submitted. >> >> On 12/5/14, Erik Weathers wrote: >> > Thanks for the response Susheel. But this setting is required for all >> > topologies, I need to set it globally so that I don't have to harass >> every >> > team into modifying their topologies with that configuration. Also, >> later >> > when netty is presumably fixed I don't want to force people to rewrite >> > their topologies to switch back from zmq to netty. >> > >> > - Erik >> > >> > On Thu, Dec 4, 2014 at 10:08 PM, Susheel Kumar Gadalay >> > >> > wrote: >> > >> >> If this is required for one topology better to set it in the Config >> >> object. >> >> >> >> Config config = new Config(); >> >> config.put(Config.storm.messaging.transport: >> >> "backtype.storm.messaging.zmq"); >> >> >> >> >> >> On 12/5/14, Erik Weathers wrote: >> >> > Can someone please clarify for me how you properly specify >> >> worker-specific >> >> > configuration in storm.yaml? >> >> > >> >> > e.g., I'm making a change to use zmq as the transport (due to a >> >> tangential >> >> > problem with netty), and it's unclear where to make this change: >> >> > >> >> > - storm.messaging.transport: backtype.storm.messaging.zmq >> >> > >> >> > I assumed that I needed to change storm.yaml on the worker machine, >> but >> >> it >> >> > seems that the change only shows up in the nimbus UI when I modify >> >> > storm.yaml on the nimbus. >> >> > >> >> > So I'm assuming that the config options you specify in storm.yaml on >> >> nimbus >> >> > propagated to the storm workers, please correct me if that's wrong. >> >> > >> >> > But what if you make a change to the configs in the nimbus and then >> >> restart >> >> > it with the topology still running? i.e., will that change be >> >> > propagated >> >> > to the existing topology workers somehow? (In the case of changing >> the >> >> > messaging transport, I assume that would mean restarting the workers >> to >> >> use >> >> > the alternative transport layer.) >> >> > >> >> > Thanks! >> >> > >> >> > - Erik >> >> > >> >> > P.S., I am well aware of the other stuff you need to do to get zmq >> >> working >> >> > in storm-0.9.1+; I just don't want to force all of our topologies to >> >> > specify the above configuration knob. >> >> > >> >> > P.P.S., I'm using storm-0.9.3 >> >> > >> >> >> > >> >