spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ram kumar <ramkumarro...@gmail.com>
Subject Re: spark streaming - checkpoint
Date Mon, 29 Jun 2015 07:54:00 GMT
on using yarn-cluster, it works good

On Mon, Jun 29, 2015 at 12:07 PM, ram kumar <ramkumarrock3@gmail.com> wrote:

> SPARK_CLASSPATH=$CLASSPATH:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/*
> in spark-env.sh
>
> I think i am facing the same issue
> https://issues.apache.org/jira/browse/SPARK-6203
>
>
>
> On Mon, Jun 29, 2015 at 11:38 AM, ram kumar <ramkumarrock3@gmail.com>
> wrote:
>
>> I am using Spark 1.2.0.2.2.0.0-82 (git revision de12451) built for Hadoop
>> 2.6.0.2.2.0.0-2041
>>
>> 1) SPARK_CLASSPATH not set
>> 2) spark.executor.extraClassPath not set
>>
>> should I upgrade my version to 1.3 and check
>>
>> On Sat, Jun 27, 2015 at 1:07 PM, Tathagata Das <tdas@databricks.com>
>> wrote:
>>
>>> Do you have SPARK_CLASSPATH set in both cases? Before and after
>>> checkpoint? If yes, then you should not be using SPARK_CLASSPATH, it has
>>> been deprecated since Spark 1.0 because of its ambiguity.
>>> Also where do you have spark.executor.extraClassPath set? I dont see it
>>> in the spark-submit command.
>>>
>>> On Fri, Jun 26, 2015 at 6:05 AM, ram kumar <ramkumarrock3@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> ---------------------------------------------
>>>>
>>>> JavaStreamingContext ssc = new JavaStreamingContext(conf, new
>>>> Duration(10000));
>>>>         ssc.checkpoint(checkPointDir);
>>>>
>>>> JavaStreamingContextFactory factory = new JavaStreamingContextFactory()
>>>> {
>>>>             public JavaStreamingContext create() {
>>>>                 return createContext(checkPointDir, outputDirectory);
>>>>             }
>>>>
>>>>         };
>>>>         JavaStreamingContext ssc =
>>>> JavaStreamingContext.getOrCreate(checkPointDir, factory);
>>>>
>>>> ----------------------------------------------------
>>>>
>>>> *first time, i run this. It work fine.*
>>>>
>>>> *but, second time. it shows following error.*
>>>> *i deleted the checkpoint path and then it works.*
>>>>
>>>> ---------------------------------------------------
>>>> [user@h7 ~]$ spark-submit --jars /home/user/examples-spark-jar.jar
>>>> --conf spark.driver.allowMultipleContexts=true --class com.spark.Pick
>>>> --master yarn-client --num-executors 10 --executor-cores 1 SNAPSHOT.jar
>>>> Spark assembly has been built with Hive, including Datanucleus jars on
>>>> classpath
>>>> 2015-06-26 12:43:42,981 WARN  [main] util.NativeCodeLoader
>>>> (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop
library
>>>> for your platform... using builtin-java classes where applicable
>>>> 2015-06-26 12:43:44,246 WARN  [main] shortcircuit.DomainSocketFactory
>>>> (DomainSocketFactory.java:<init>(116)) - The short-circuit local reads
>>>> feature cannot be used because libhadoop cannot be loaded.
>>>>
>>>> This is deprecated in Spark 1.0+.
>>>>
>>>> Please instead use:
>>>>  - ./spark-submit with --driver-class-path to augment the driver
>>>> classpath
>>>>  - spark.executor.extraClassPath to augment the executor classpath
>>>>
>>>> Exception in thread "main" org.apache.spark.SparkException: Found both
>>>> spark.executor.extraClassPath and SPARK_CLASSPATH. Use only the former.
>>>>     at
>>>> org.apache.spark.SparkConf$$anonfun$validateSettings$6$$anonfun$apply$7.apply(SparkConf.scala:334)
>>>>     at
>>>> org.apache.spark.SparkConf$$anonfun$validateSettings$6$$anonfun$apply$7.apply(SparkConf.scala:332)
>>>>     at scala.collection.immutable.List.foreach(List.scala:318)
>>>>     at
>>>> org.apache.spark.SparkConf$$anonfun$validateSettings$6.apply(SparkConf.scala:332)
>>>>     at
>>>> org.apache.spark.SparkConf$$anonfun$validateSettings$6.apply(SparkConf.scala:320)
>>>>     at scala.Option.foreach(Option.scala:236)
>>>>     at org.apache.spark.SparkConf.validateSettings(SparkConf.scala:320)
>>>>     at org.apache.spark.SparkContext.<init>(SparkContext.scala:178)
>>>>     at
>>>> org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:118)
>>>>     at
>>>> org.apache.spark.streaming.StreamingContext$$anonfun$getOrCreate$1.apply(StreamingContext.scala:561)
>>>>     at
>>>> org.apache.spark.streaming.StreamingContext$$anonfun$getOrCreate$1.apply(StreamingContext.scala:561)
>>>>     at scala.Option.map(Option.scala:145)
>>>>     at
>>>> org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:561)
>>>>     at
>>>> org.apache.spark.streaming.api.java.JavaStreamingContext$.getOrCreate(JavaStreamingContext.scala:566)
>>>>     at
>>>> org.apache.spark.streaming.api.java.JavaStreamingContext.getOrCreate(JavaStreamingContext.scala)
>>>>     at
>>>> com.orzota.kafka.kafka.TotalPicsWithScore.main(TotalPicsWithScore.java:159)
>>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>     at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>     at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>     at java.lang.reflect.Method.invoke(Method.java:606)
>>>>     at
>>>> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:360)
>>>>     at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:76)
>>>>     at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>> [user@h7 ~]
>>>>
>>>> ----------------------------------------------
>>>>
>>>> *can anyone help me with it*
>>>>
>>>>
>>>> *thanks*
>>>>
>>>
>>>
>>
>

Mime
View raw message