spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tamas Jambor <jambo...@gmail.com>
Subject Re: spark context not picking up default hadoop filesystem
Date Mon, 26 Jan 2015 16:06:44 GMT
thanks for the reply. I have tried to add SPARK_CLASSPATH, I got a warning
that it was deprecated (didn't solve the problem), also tried to run with
--driver-class-path, which did not work either. I am trying this locally.



On Mon Jan 26 2015 at 15:04:03 Akhil Das <akhil@sigmoidanalytics.com> wrote:

> You can also trying adding the core-site.xml in the SPARK_CLASSPATH, btw
> are you running the application locally? or in standalone mode?
>
> Thanks
> Best Regards
>
> On Mon, Jan 26, 2015 at 7:37 PM, jamborta <jamborta@gmail.com> wrote:
>
>> hi all,
>>
>> I am trying to create a spark context programmatically, using
>> org.apache.spark.deploy.SparkSubmit. It all looks OK, except that the
>> hadoop
>> config that is created during the process is not picking up core-site.xml,
>> so it defaults back to the local file-system. I have set HADOOP_CONF_DIR
>> in
>> spark-env.sh, also core-site.xml in in the conf folder. The whole thing
>> works if it is executed through spark shell.
>>
>> Just wondering where spark is picking up the hadoop config path from?
>>
>> many thanks,
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/spark-context-not-picking-up-default-hadoop-filesystem-tp21368.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>

Mime
View raw message