storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cody A. Ray" <cody.a....@gmail.com>
Subject Re: Machine specs
Date Thu, 01 May 2014 19:18:42 GMT
I hate to give this answer, but I think it really depends on your
application. If you're doing distributed machine learning or video
compression or something that's CPU heavy, then it'll be CPU heavy. If
you're doing pre-aggregation or rolling windows or other CPU-light
analysis, you're more likely to be memory- or network- bound.

Or people might just be scaling horizontally across a lot of cheap worker
nodes rather than fewer nodes with a lot of CPUs. :)

-Cody


On Thu, May 1, 2014 at 11:57 AM, Software Dev <static.void.dev@gmail.com>wrote:

> Seems like all of these setups involve a small number of CPU's??? Does
> storm typically require more RAM than CPU.. ie which is usually the
> bottleneck?
>
> On Wed, Apr 30, 2014 at 8:54 PM, Michael Rose <michael@fullcontact.com>
> wrote:
> > In AWS, we're fans of c1.xlarges, m3.xlarges, and c3.2xlarges, but have
> seen
> > Storm successfully run on cheaper hardware.
> >
> > Our Nimbus server is usually bored on a m1.large.
> >
> > Michael Rose (@Xorlev)
> > Senior Platform Engineer, FullContact
> > michael@fullcontact.com
> >
> >
> >
> > On Wed, Apr 30, 2014 at 9:48 PM, Cody A. Ray <cody.a.ray@gmail.com>
> wrote:
> >>
> >> We use m1.larges in EC2 for both nimbus and supervisor machines (though
> >> the m1 family have been deprecated in favor of m3). Our use case is to
> do
> >> some pre-aggregation before persisting the data in a store. (The main
> >> bottleneck in this setup is the downstream datastore, but memory is the
> >> primary constraint on the worker machines due to the in-memory cache
> which
> >> wraps the trident state.)
> >>
> >> For what its worth, Infochimps suggests c1.xlarge or m3.xlarge machines.
> >>
> >> Using the Amazon cloud machines as a reference, we like to use either
> the
> >> c1.xlarge machines (7GB ram, 8 cores, $424/month, giving the highest
> >> CPU-performance-per-dollar) or the m3.xlargemachines (15 GB ram, 4
> cores,
> >> $365/month, the best balance of CPU-per-dollar and RAM-per-dollar). You
> >> shouldn’t use fewer than four worker machines in production, so if your
> >> needs are modest feel free to downsize the hardware accordingly.
> >>
> >> Not sure what others would recommend.
> >>
> >> -Cody
> >>
> >>
> >> On Wed, Apr 30, 2014 at 5:57 PM, Software Dev <
> static.void.dev@gmail.com>
> >> wrote:
> >>>
> >>> What kind of specs are we looking at for
> >>>
> >>> 1) Nimbus
> >>> 2) Workers
> >>>
> >>> Any recommendations?
> >>
> >>
> >>
> >>
> >> --
> >> Cody A. Ray, LEED AP
> >> cody.a.ray@gmail.com
> >> 215.501.7891
> >
> >
>



-- 
Cody A. Ray, LEED AP
cody.a.ray@gmail.com
215.501.7891

Mime
View raw message