spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chang Lim <chang...@gmail.com>
Subject DStream are not processed after upgrade to Spark 1.0
Date Sat, 14 Jun 2014 18:37:45 GMT
Hi All,

I've some Streaming code in Java that works on 0.9.1.  After upgrading to
1.0 (with fix to minor API changes) DStream does not seem to be executing.
The tasks got killed in 1 second by the worker.  Any idea what is causing
it?

The worker log file is not logging my debug statements.  The following
console output seems to be executing something but my debug logs and
stream.print() is not being logged.  I see my console println statement on
the context being started.  I am using Spark standalone cluster.

Any help is much appreciated.

Console Log:
===========
14/06/13 22:54:07 INFO AppClient$ClientActor: Connecting to master
spark://10.206.133.73:7077...
14/06/13 22:54:07 INFO SparkDeploySchedulerBackend: Connected to Spark
cluster with app ID app-20140613225220-0001
14/06/13 22:54:07 INFO AppClient$ClientActor: Executor added:
app-20140613225220-0001/0 on worker-20140613223732-tahiti-ins.xxx.com-46311
(tahiti-ins.xxx.com:46311) with 1 cores
14/06/13 22:54:07 INFO SparkDeploySchedulerBackend: Granted executor ID
app-20140613225220-0001/0 on hostPort tahiti-ins.xxx.com:46311 with 1 cores,
512.0 MB RAM
14/06/13 22:54:07 INFO AppClient$ClientActor: Executor updated:
app-20140613225220-0001/0 is now RUNNING
Reading CSV File
SSC Context Start
14/06/13 22:54:08 INFO StateDStream: Checkpoint interval automatically set
to 10000 ms
14/06/13 22:54:08 INFO StateDStream: Checkpoint interval automatically set
to 10000 ms
14/06/13 22:54:08 INFO MappedDStream: Duration for remembering RDDs set to
20000 ms for org.apache.spark.streaming.dstream.MappedDStream@1158876
14/06/13 22:54:08 INFO QueueInputDStream: Duration for remembering RDDs set
to 20000 ms for org.apache.spark.streaming.dstream.QueueInputDStream@b2100c
14/06/13 22:54:08 INFO TransformedDStream: Duration for remembering RDDs set
to 20000 ms for org.apache.spark.streaming.dstream.TransformedDStream@3ba710
14/06/13 22:54:08 INFO ShuffledDStream: Duration for remembering RDDs set to
20000 ms for org.apache.spark.streaming.dstream.ShuffledDStream@550ea2
14/06/13 22:54:08 INFO MappedDStream: Duration for remembering RDDs set to
20000 ms for org.apache.spark.streaming.dstream.MappedDStream@33c98c
14/06/13 22:54:08 INFO ShuffledDStream: Duration for remembering RDDs set to
20000 ms for org.apache.spark.streaming.dstream.ShuffledDStream@1f50b8f
14/06/13 22:54:08 INFO MappedDStream: Duration for remembering RDDs set to
20000 ms for org.apache.spark.streaming.dstream.MappedDStream@ca0145
. . .
14/06/13 22:54:08 INFO ShuffledDStream: Initialized and validated
org.apache.spark.streaming.dstream.ShuffledDStream@1f50b8f
14/06/13 22:54:08 INFO TransformedDStream: Slide time = 10000 ms
14/06/13 22:54:08 INFO TransformedDStream: Storage level =
StorageLevel(false, false, false, false, 1)
14/06/13 22:54:08 INFO TransformedDStream: Checkpoint interval = null
14/06/13 22:54:08 INFO TransformedDStream: Remember duration = 20000 ms
14/06/13 22:54:08 INFO TransformedDStream: Initialized and validated
org.apache.spark.streaming.dstream.TransformedDStream@3ba710
14/06/13 22:54:08 INFO StateDStream: Slide time = 10000 ms
14/06/13 22:54:08 INFO StateDStream: Storage level = StorageLevel(false,
true, false, false, 1)
14/06/13 22:54:08 INFO StateDStream: Checkpoint interval = 10000 ms
14/06/13 22:54:08 INFO StateDStream: Remember duration = 20000 ms
14/06/13 22:54:08 INFO StateDStream: Initialized and validated
org.apache.spark.streaming.dstream.StateDStream@140bbad
14/06/13 22:54:08 INFO ForEachDStream: Slide time = 10000 ms
14/06/13 22:54:08 INFO ForEachDStream: Storage level = StorageLevel(false,
false, false, false, 1)
14/06/13 22:54:08 INFO ForEachDStream: Checkpoint interval = null
14/06/13 22:54:08 INFO ForEachDStream: Remember duration = 10000 ms
14/06/13 22:54:08 INFO ForEachDStream: Initialized and validated
org.apache.spark.streaming.dstream.ForEachDStream@cf3c09
14/06/13 22:54:08 INFO RecurringTimer: Started timer for JobGenerator at
time 1402725250000
14/06/13 22:54:08 INFO JobGenerator: Started JobGenerator at 1402725250000
ms
14/06/13 22:54:08 INFO JobScheduler: Started JobScheduler



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/DStream-are-not-processed-after-upgrade-to-Spark-1-0-tp7627.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Mime
View raw message