spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <>
Subject Re: Unable to load native-hadoop library for your platform
Date Wed, 05 Aug 2015 16:52:11 GMT

> On 4 Aug 2015, at 12:26, Sean Owen <> wrote:
> Oh good point, does the Windows integration need native libs for
> POSIX-y file system access? I know there are some binaries shipped for
> this purpose but wasn't sure if that's part of what's covered in the
> native libs message.

On unix its a warning that you'll get slower access to bits of the system and no posix-ish
support, on windows its more serious (and should be reported more seriously)

It's used for various low-level things, and currently Hadoop fails somewhat messily when its
not there. That's even for things like MiniYarnCluster, which is a real annoyance.

I have a snapshot of the Hadoop 2.6 windows binaries if people want them; I should stick up
a copy of the 2.7.1 ones too

> On Tue, Aug 4, 2015 at 6:01 PM, Steve Loughran <> wrote:
>> Think it may be needed on Windows, certainly if you start trying to work with local
>>> On 4 Aug 2015, at 00:34, Sean Owen <> wrote:
>>> It won't affect you if you're not actually running Hadoop. But it's
>>> mainly things like Snappy/LZO compression which are implemented as
>>> native libraries under the hood.
>> There's a lot more in those native libs, primarily to bypass bits missing from the
java APIs (FS permissions) and to add new features (encryption, soon erasure coding).
>> The Hadoop file:// FS uses it on windows, at least for now

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message