spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Ash <and...@andrewash.com>
Subject Re: Contributing Spark Infrastructure Configuration Docs
Date Thu, 05 Jun 2014 23:09:58 GMT
I would appreciate seeing the specs you came up with as well but don't need
to particularly quickly.  I'll wait until seeing the PR to comment on the
specifics, but have some questions about the thought process that went into
configuring the hardware.

Is the idea to see how you spec'd out memory/disk/processor for each
physical machine?  How did you optimize the ratios between the three for
Spark specifically?  Were you able to test this configuration against
others to optimize for price per performance?

Sorry for the barrage of questions -- I know talking hardware tends to
bring out the crazy in people.

Cheers,
Andrew


On Thu, Jun 5, 2014 at 4:04 PM, Krishna Sankar <ksankar42@gmail.com> wrote:

> Stephen,
>     We are working thru Dell configurations; would be happy to review your
> diagrams and offer feedback from our experience. Let me know the URLs.
> Cheers
> <k/>
>
>
> On Thu, Jun 5, 2014 at 2:51 PM, Stephen Watt <swatt@redhat.com> wrote:
>
> > Hi Folks
> >
> > My name is Steve Watt and I work in the CTO Office at Red Hat. I've
> > recently spent quite a bit of time designing single rack and multi-rack
> > infrastructures for Spark for our own hardware procurement at Red Hat
> and I
> > thought the diagrams and server specs for both Dell and HP would be
> useful
> > to the broader community as well. Even if folks don't want to go with my
> > exact design, having the designs as a starting point should save  quite a
> > bit of time. I think I can fold this quite easily into
> > http://spark.apache.org/docs/latest/hardware-provisioning.html
> >
> > However, before submitting a pull request for the modified page I thought
> > I'd send this note up front and see if anyone wanted to provide some
> > feedback first. If there is none, I'll submit the pull request early next
> > week.
> >
> > Regards
> > Steve Watt
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message