spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: issue while creating spark context
Date Tue, 24 Mar 2015 13:08:25 GMT
Its in your local file system, not in hdfs.

Thanks
Best Regards

On Tue, Mar 24, 2015 at 6:25 PM, Sachin Singh <sachin.shashi@gmail.com>
wrote:

> hi,
> I can see required permission is granted for this directory as under,
>
>  hadoop dfs -ls /user/spark
> DEPRECATED: Use of this script to execute hdfs command is deprecated.
> Instead use the hdfs command for it.
>
> Found 1 items
> *drwxrwxrwt   - spark spark          0 2015-03-20 01:04
> /user/spark/applicationHistory*
>
> regards
> Sachin
>
>
> On Tue, Mar 24, 2015 at 6:13 PM, Akhil Das <akhil@sigmoidanalytics.com>
> wrote:
>
>> write permission as its clearly saying:
>>
>> java.io.IOException:* Error in creating log directory:*
>> file:*/user/spark/*applicationHistory/application_1427194309307_0005
>>
>> Thanks
>> Best Regards
>>
>> On Tue, Mar 24, 2015 at 6:08 PM, Sachin Singh <sachin.shashi@gmail.com>
>> wrote:
>>
>>> Hi Akhil,
>>> thanks for your quick reply,
>>> I would like to request please elaborate i.e. what kind of permission
>>> required ..
>>>
>>> thanks in advance,
>>>
>>> Regards
>>> Sachin
>>>
>>> On Tue, Mar 24, 2015 at 5:29 PM, Akhil Das <akhil@sigmoidanalytics.com>
>>> wrote:
>>>
>>>> Its an IOException, just make sure you are having the correct
>>>> permission over */user/spark* directory.
>>>>
>>>> Thanks
>>>> Best Regards
>>>>
>>>> On Tue, Mar 24, 2015 at 5:21 PM, sachin Singh <sachin.shashi@gmail.com>
>>>> wrote:
>>>>
>>>>> hi all,
>>>>> all of sudden I getting below error when I am submitting spark job
>>>>> using
>>>>> master as yarn its not able to create spark context,previously working
>>>>> fine,
>>>>> I am using CDH5.3.1 and creating javaHiveContext
>>>>> spark-submit --jars
>>>>>
>>>>> ./analiticlibs/mysql-connector-java-5.1.17.jar,./analiticlibs/log4j-1.2.17.jar
>>>>> --master yarn --class myproject.com.java.jobs.Aggregationtask
>>>>> sparkjob-1.0.jar
>>>>>
>>>>> error message-
>>>>> java.io.IOException: Error in creating log directory:
>>>>> file:/user/spark/applicationHistory/application_1427194309307_0005
>>>>>         at
>>>>> org.apache.spark.util.FileLogger.createLogDir(FileLogger.scala:133)
>>>>>         at org.apache.spark.util.FileLogger.start(FileLogger.scala:115)
>>>>>         at
>>>>>
>>>>> org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:74)
>>>>>         at org.apache.spark.SparkContext.<init>(SparkContext.scala:353)
>>>>>         at
>>>>>
>>>>> org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:61)
>>>>>         at
>>>>>
>>>>> myproject.com.java.core.SparkAnaliticEngine.getJavaSparkContext(SparkAnaliticEngine.java:77)
>>>>>         at
>>>>>
>>>>> myproject.com.java.core.SparkAnaliticTable.evmyprojectate(SparkAnaliticTable.java:108)
>>>>>         at
>>>>>
>>>>> myproject.com.java.core.SparkAnaliticEngine.evmyprojectateAnaliticTable(SparkAnaliticEngine.java:55)
>>>>>         at
>>>>>
>>>>> myproject.com.java.core.SparkAnaliticEngine.evmyprojectateAnaliticTable(SparkAnaliticEngine.java:65)
>>>>>         at
>>>>>
>>>>> myproject.com.java.jobs.CustomAggregationJob.main(CustomAggregationJob.java:184)
>>>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>         at
>>>>>
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>         at
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>         at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>         at
>>>>> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
>>>>>         at
>>>>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
>>>>>         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/issue-while-creating-spark-context-tp22196.html
>>>>> Sent from the Apache Spark User List mailing list archive at
>>>>> Nabble.com.
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message