spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@hortonworks.com>
Subject Re: Unable to load native-hadoop library for your platform
Date Tue, 04 Aug 2015 17:01:47 GMT
Think it may be needed on Windows, certainly if you start trying to work with local files.



> On 4 Aug 2015, at 00:34, Sean Owen <sowen@cloudera.com> wrote:
> 
> It won't affect you if you're not actually running Hadoop. But it's
> mainly things like Snappy/LZO compression which are implemented as
> native libraries under the hood.

There's a lot more in those native libs, primarily to bypass bits missing from the java APIs
(FS permissions) and to add new features (encryption, soon erasure coding).

The Hadoop file:// FS uses it on windows, at least for now

> Spark doesn't necessarily use these
> anyway; it's from the Hadoop libs.
> 
> On Tue, Aug 4, 2015 at 8:30 AM, Deepesh Maheshwari
> <deepesh.maheshwari17@gmail.com> wrote:
>> Can you elaborate about the things this native library covering.
>> One you mentioned accelerated compression.
>> 
>> It would be very helpful if you can give any useful to link to read more
>> about it.
>> 
>> On Tue, Aug 4, 2015 at 12:56 PM, Sean Owen <sowen@cloudera.com> wrote:
>>> 
>>> You can ignore it entirely. It just means you haven't installed and
>>> configured native libraries for things like accelerated compression,
>>> but it has no negative impact otherwise.
>>> 
>>> On Tue, Aug 4, 2015 at 8:11 AM, Deepesh Maheshwari
>>> <deepesh.maheshwari17@gmail.com> wrote:
>>>> Hi,
>>>> 
>>>> When i run the spark locally on windows it gives below hadoop library
>>>> error.
>>>> I am using below spark version.
>>>> 
>>>> <dependency>
>>>>            <groupId>org.apache.spark</groupId>
>>>>            <artifactId>spark-core_2.10</artifactId>
>>>>            <version>1.4.1</version>
>>>>        </dependency>
>>>> 
>>>> 
>>>> 2015-08-04 12:22:23,463  WARN
>>>> (org.apache.hadoop.util.NativeCodeLoader:62) -
>>>> Unable to load native-hadoop library for your platform... using
>>>> builtin-java
>>>> classes where applicable
>>>> 
>>>> Tried to find it on internet but not able to find exact root cause.
>>>> Please let me know what is it, why it is giving warning and how can i
>>>> resolve it.
>>>> 
>>>> Thanks,
>>>> Deepesh
>> 
>> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message