spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@hortonworks.com>
Subject Re: Reading from Amazon S3
Date Tue, 03 May 2016 10:59:39 GMT

On 2 May 2016, at 19:24, Gourav Sengupta <gourav.sengupta@gmail.com<mailto:gourav.sengupta@gmail.com>>
wrote:

Jorn,

what aspects are you speaking about ?

My response was absolutely pertinent to Jinan because he will not even face the problem if
he used Scala. So it was along the lines of helping a person to learn fishing that giving
him a fish.

Sorry: it will. This is a compatibility issue with Hadoop s3a as implemented in hadoop-aws.jar
and amazon s3 libraries, something you can replicate in ScalaContext.textFile()

This is a classpath problem, not a language issue


And by the way your blinkered and biased response missed the fact that SPARK WAS WRITTEN AND
IS WRITTEN IN SCALA.

And runs in the JVM, including much of the core ASF libraries, google guava, Hadoop core,
Zookeeper, etc. Oh, and hadoop has some C native libraries which you should have for Unix
performance, and will need for Windows.  There are bits of bash and python around, and even
groovy in some of the spark-assembly JARs, though that's an accident which has been corrected
in recent versions.

Languages are tools: the more you know, the more tools you have at your disposal.

Now, if you really want to make the problem go away, a patch for https://issues.apache.org/jira/browse/HADOOP-13062
would be nice; I promise I will review it

Mime
View raw message