spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@hortonworks.com>
Subject Re: Unable to load native-hadoop library for your platform
Date Wed, 05 Aug 2015 16:52:11 GMT

> On 4 Aug 2015, at 12:26, Sean Owen <sowen@cloudera.com> wrote:
> 
> Oh good point, does the Windows integration need native libs for
> POSIX-y file system access? I know there are some binaries shipped for
> this purpose but wasn't sure if that's part of what's covered in the
> native libs message.

On unix its a warning that you'll get slower access to bits of the system and no posix-ish
support, on windows its more serious (and should be reported more seriously)


It's used for various low-level things, and currently Hadoop fails somewhat messily when its
not there. That's even for things like MiniYarnCluster, which is a real annoyance.

I have a snapshot of the Hadoop 2.6 windows binaries if people want them; I should stick up
a copy of the 2.7.1 ones too
https://github.com/steveloughran/clusterconfigs/tree/master/clusters/morzine/hadoop_home/


> 
> On Tue, Aug 4, 2015 at 6:01 PM, Steve Loughran <stevel@hortonworks.com> wrote:
>> Think it may be needed on Windows, certainly if you start trying to work with local
files.
>> 
>> 
>>> On 4 Aug 2015, at 00:34, Sean Owen <sowen@cloudera.com> wrote:
>>> 
>>> It won't affect you if you're not actually running Hadoop. But it's
>>> mainly things like Snappy/LZO compression which are implemented as
>>> native libraries under the hood.
>> 
>> There's a lot more in those native libs, primarily to bypass bits missing from the
java APIs (FS permissions) and to add new features (encryption, soon erasure coding).
>> 
>> The Hadoop file:// FS uses it on windows, at least for now
>> 


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message