From dev-return-15698-apmail-spark-dev-archive=spark.apache.org@spark.apache.org Sun Nov 1 22:33:08 2015 Return-Path: X-Original-To: apmail-spark-dev-archive@minotaur.apache.org Delivered-To: apmail-spark-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2519C18602 for ; Sun, 1 Nov 2015 22:33:08 +0000 (UTC) Received: (qmail 69017 invoked by uid 500); 1 Nov 2015 22:33:06 -0000 Delivered-To: apmail-spark-dev-archive@spark.apache.org Received: (qmail 68909 invoked by uid 500); 1 Nov 2015 22:33:06 -0000 Mailing-List: contact dev-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@spark.apache.org Received: (qmail 68898 invoked by uid 99); 1 Nov 2015 22:33:05 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Nov 2015 22:33:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 6EAA71A273D for ; Sun, 1 Nov 2015 22:33:05 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.001 X-Spam-Level: X-Spam-Status: No, score=0.001 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=eecs_berkeley_edu.20150623.gappssmtp.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id RceO0G-3z1M7 for ; Sun, 1 Nov 2015 22:32:52 +0000 (UTC) Received: from mail-lf0-f47.google.com (mail-lf0-f47.google.com [209.85.215.47]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 6728320FF0 for ; Sun, 1 Nov 2015 22:32:51 +0000 (UTC) Received: by lffz202 with SMTP id z202so54544946lff.3 for ; Sun, 01 Nov 2015 14:32:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=eecs_berkeley_edu.20150623.gappssmtp.com; s=20150623; h=mime-version:reply-to:in-reply-to:references:from:date:message-id :subject:to:cc:content-type:content-transfer-encoding; bh=jllfLA6NH2LHH8+DgV2I2YEKoFpTDLo6rf2ql9RLu1g=; b=dsS4ckDRD3b5MMwrvNrATcvF3XfWNxUyPBQ517s9hK9n+Ka+wgnAVrbFLNRP0eXCIW hdq9C6Hu1CUIT6EqONpsGMg13NHRUff8t9YV47tF71nL1o3CYF6djVua3p/4mWCy0WmO rbbAZeTbho8uzh/7yclVTI2e8Kxco/j0S9siXDD6BDUWuNRMl7wj8mIGBuffyrrpcSYY NXcJcORh6y/DbAHAGX1YelA3YElu+fcePqb10OXqlY9d3k4eeJshm4q7dj69q9nsA1D+ X3m5MfNtiw8Q7o43695PzL7XkePS/e0hjHrO8G5A5frPPLhk24mEqhP+KNgGF5OHtyYE SOgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :from:date:message-id:subject:to:cc:content-type :content-transfer-encoding; bh=jllfLA6NH2LHH8+DgV2I2YEKoFpTDLo6rf2ql9RLu1g=; b=guJZ7EraFR3oumXmrxuslXi14vE+JpnUYIUN+jmtsrCx/EeD3KdH6W/tyvoLH+/hd2 4oNnd2TNcjf6kx2pYJgOIkZv6fquBsa0rUJymvGcuv5UWA7aljqg6jlQX9is0AsM0YRY NSvyu/PxA289Eiz/tsrK5eRvND+69yaVxNUseMkBnF4wbIR13qOmaNwYvAanLGrHEry2 Mzbpz+YrRaF+hp1eST9xs/rR0F0NfsQbCXDWxBWAaEK0VJXIxULqpPtDwzJRObmF/IWi nSILwu+OvuWL24e2Eoq4nbBJWCqLo74k3ad/APEeQmEWrMrhyfslx1N2WWSuI2iO38/F rwEg== X-Gm-Message-State: ALoCoQm3E4YVuHey4BlSrSog39okTJ7m7aHdnaAi7XYY2sIi4WHgsyGRlpQgi+Cx88PsKtOxRh5H X-Received: by 10.25.16.73 with SMTP id f70mr5726680lfi.21.1446417169446; Sun, 01 Nov 2015 14:32:49 -0800 (PST) MIME-Version: 1.0 Reply-To: shivaram@eecs.berkeley.edu Received: by 10.25.196.144 with HTTP; Sun, 1 Nov 2015 14:32:30 -0800 (PST) In-Reply-To: References: <92065050-343B-4C41-B4DF-08A7676EF038@hortonworks.com> From: Shivaram Venkataraman Date: Sun, 1 Nov 2015 14:32:30 -0800 Message-ID: Subject: Re: Downloading Hadoop from s3://spark-related-packages/ To: Nicholas Chammas Cc: Shivaram Venkataraman , Steve Loughran , Spark dev list Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Sun, Nov 1, 2015 at 2:16 PM, Nicholas Chammas wrote: > OK, I=E2=80=99ll focus on the Apache mirrors going forward. > > The problem with the Apache mirrors, if I am not mistaken, is that you > cannot use a single URL that automatically redirects you to a working mir= ror > to download Hadoop. You have to pick a specific mirror and pray it doesn= =E2=80=99t > disappear tomorrow. > > They don=E2=80=99t go away, especially http://mirror.ox.ac.uk , and in th= e us the > apache.osuosl.org, osu being a where a lot of the ASF servers are kept. > > So does Apache offer no way to query a URL and automatically get the clos= est > working mirror? If I=E2=80=99m installing HDFS onto servers in various EC= 2 regions, > the best mirror will vary depending on my location. > Not sure if this is officially documented somewhere but if you pass '&asjson=3D1' you will get back a JSON which has a 'preferred' field set to the closest mirror. Shivaram > Nick > > > On Sun, Nov 1, 2015 at 12:25 PM Shivaram Venkataraman > wrote: >> >> I think that getting them from the ASF mirrors is a better strategy in >> general as it'll remove the overhead of keeping the S3 bucket up to >> date. It works in the spark-ec2 case because we only support a limited >> number of Hadoop versions from the tool. FWIW I don't have write >> access to the bucket and also haven't heard of any plans to support >> newer versions in spark-ec2. >> >> Thanks >> Shivaram >> >> On Sun, Nov 1, 2015 at 2:30 AM, Steve Loughran >> wrote: >> > >> > On 1 Nov 2015, at 03:17, Nicholas Chammas >> > wrote: >> > >> > https://s3.amazonaws.com/spark-related-packages/ >> > >> > spark-ec2 uses this bucket to download and install HDFS on clusters. I= s >> > it >> > owned by the Spark project or by the AMPLab? >> > >> > Anyway, it looks like the latest Hadoop install available on there is >> > Hadoop >> > 2.4.0. >> > >> > Are there plans to add newer versions of Hadoop for use by spark-ec2 a= nd >> > similar tools, or should we just be getting that stuff via an Apache >> > mirror? >> > The latest version is 2.7.1, by the way. >> > >> > >> > you should be grabbing the artifacts off the ASF and then verifying >> > their >> > SHA1 checksums as published on the ASF HTTPS web site >> > >> > >> > The problem with the Apache mirrors, if I am not mistaken, is that you >> > cannot use a single URL that automatically redirects you to a working >> > mirror >> > to download Hadoop. You have to pick a specific mirror and pray it >> > doesn't >> > disappear tomorrow. >> > >> > >> > They don't go away, especially http://mirror.ox.ac.uk , and in the us >> > the >> > apache.osuosl.org, osu being a where a lot of the ASF servers are kept= . >> > >> > full list with availability stats >> > >> > http://www.apache.org/mirrors/ >> > >> > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org For additional commands, e-mail: dev-help@spark.apache.org