spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marco Mistroni <mmistr...@gmail.com>
Subject Re: Hive error when starting up spark-shell in 1.5.2
Date Thu, 24 Dec 2015 19:51:18 GMT
No luck.
But two updates:
1. i have downloaded spark-1.4.1 and everything works fine, i dont see any
error
2. i have added the following XML file to spark's 1.5.2  conf directory and
now i got the following error

aused by: java.lang.RuntimeException: The root scratch dir:
c:/Users/marco/tmp on HDFS should be writable. Current permissions are:
rwx---rwx

I will have to play around with windows permissions to allow spark to use
that directory

kr
 marco




On Sun, Dec 20, 2015 at 5:15 PM, Marco Mistroni <mmistroni@gmail.com> wrote:

> Thanks Chris will give it a go and report back.
> Bizarrely if I start the pyspark shell I don't see any issues
> Kr
> Marco
> On 20 Dec 2015 5:02 pm, "Chris Fregly" <chris@fregly.com> wrote:
>
>> hopping on a plane, but check the hive-site.xml that's in your spark/conf
>> directory (or should be, anyway).  I believe you can change the root path
>> thru this mechanism.
>>
>> if not, this should give you more info google on.
>>
>> let me know as this comes up a fair amount.
>>
>> > On Dec 19, 2015, at 4:58 PM, Marco Mistroni <mmistroni@gmail.com>
>> wrote:
>> >
>> > HI all
>> >  posting again this as i was experiencing this error also under 1.5.1
>> > I am running spark 1.5.2 on a Windows 10 laptop (upgraded from Windows
>> 8)
>> > When i launch spark-shell i am getting this exception, presumably
>> becaus ei hav eno
>> > admin right to /tmp directory on my latpop (windows 8-10 seems very
>> restrictive)
>> >
>> > java.lang.RuntimeException: java.lang.RuntimeException: The root
>> scratch dir: /tmp/hive on HDFS should be writable. Current permissions are:
>> ---------
>> >         at
>> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
>> >         at
>> org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:171)
>> >         at
>> org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:162)
>> >         at
>> org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:160)
>> >         at
>> org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:167)
>> >         at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>> >         at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>> >         at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> >         at
>> java.lang.reflect.Constructor.newInstance(Constructor.java:422)
>> >         at
>> org.apache.spark.repl.SparkILoop.createSQLContext(SparkILoop.scala:1028)
>> >         at $iwC$$iwC.<init>(<console>:9)
>> >         at $iwC.<init>(<console>:18)
>> >         at <init>(<console>:20)
>> >         at .<init>(<console>:24)
>> >         at .<clinit>(<console>)
>> >         at .<init>(<console>:7)
>> >         at .<clinit>(<console>)
>> >         at $print(<console>)
>> > ......
>> > Caused by: java.lang.RuntimeException: The root scratch dir: /tmp/hive
>> on HDFS should be writable. Current permissions are: ---------
>> >         at
>> org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:612)
>> >         at
>> org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554)
>> >         at
>> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)
>> >         ... 56 more
>> >
>> > <console>:10: error: not found: value sqlContext
>> >        import sqlContext.implicits._
>> >               ^
>> > <console>:10: error: not found: value sqlContext
>> >        import sqlContext.sql
>> >
>> > I was wondering how can i configure hive to point to a different
>> directory....where i have more permissions
>> >
>> > kr
>> >  marco
>> >
>>
>

Mime
View raw message