spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "bit1129@163.com" <bit1...@163.com>
Subject Re: Spark + Kafka
Date Wed, 01 Apr 2015 07:39:38 GMT
Please make sure that you have given more cores than Receiver numbers.



 
From: James King
Date: 2015-04-01 15:21
To: user
Subject: Spark + Kafka
I have a simple setup/runtime of Kafka and Sprak.

I have a command line consumer displaying arrivals to Kafka topic. So i know messages are
being received.

But when I try to read from Kafka topic I get no messages, here are some logs below.

I'm thinking there aren't enough threads. How do i check that.

Thank you.

2015-04-01 08:56:50 INFO  JobScheduler:59 - Starting job streaming job 1427871410000 ms.0
from job set of time 1427871410000 ms
2015-04-01 08:56:50 INFO  JobScheduler:59 - Finished job streaming job 1427871410000 ms.0
from job set of time 1427871410000 ms
2015-04-01 08:56:50 INFO  JobScheduler:59 - Total delay: 0.002 s for time 1427871410000 ms
(execution: 0.000 s)
2015-04-01 08:56:50 DEBUG JobGenerator:63 - Got event ClearMetadata(1427871410000 ms)
2015-04-01 08:56:50 DEBUG DStreamGraph:63 - Clearing metadata for time 1427871410000 ms
2015-04-01 08:56:50 DEBUG ForEachDStream:63 - Clearing references to old RDDs: []
2015-04-01 08:56:50 DEBUG ForEachDStream:63 - Unpersisting old RDDs: 
2015-04-01 08:56:50 DEBUG ForEachDStream:63 - Cleared 0 RDDs that were older than 1427871405000
ms: 
2015-04-01 08:56:50 DEBUG KafkaInputDStream:63 - Clearing references to old RDDs: [1427871405000
ms -> 8]
2015-04-01 08:56:50 DEBUG KafkaInputDStream:63 - Unpersisting old RDDs: 8
2015-04-01 08:56:50 INFO  BlockRDD:59 - Removing RDD 8 from persistence list
2015-04-01 08:56:50 DEBUG BlockManagerMasterActor:50 - [actor] received message RemoveRdd(8)
from Actor[akka://sparkDriver/temp/$n]
2015-04-01 08:56:50 DEBUG BlockManagerSlaveActor:50 - [actor] received message RemoveRdd(8)
from Actor[akka://sparkDriver/temp/$o]
2015-04-01 08:56:50 DEBUG BlockManagerMasterActor:56 - [actor] handled message (0.287257 ms)
RemoveRdd(8) from Actor[akka://sparkDriver/temp/$n]
2015-04-01 08:56:50 DEBUG BlockManagerSlaveActor:63 - removing RDD 8
2015-04-01 08:56:50 INFO  BlockManager:59 - Removing RDD 8
2015-04-01 08:56:50 DEBUG BlockManagerSlaveActor:63 - Done removing RDD 8, response is 0
2015-04-01 08:56:50 DEBUG BlockManagerSlaveActor:56 - [actor] handled message (0.038047 ms)
RemoveRdd(8) from Actor[akka://sparkDriver/temp/$o]
2015-04-01 08:56:50 INFO  KafkaInputDStream:59 - Removing blocks of RDD BlockRDD[8] at createStream
at KafkaLogConsumer.java:53 of time 1427871410000 ms
2015-04-01 08:56:50 DEBUG KafkaInputDStream:63 - Cleared 1 RDDs that were older than 1427871405000
ms: 1427871405000 ms
2015-04-01 08:56:50 DEBUG DStreamGraph:63 - Cleared old metadata for time 1427871410000 ms
2015-04-01 08:56:50 INFO  ReceivedBlockTracker:59 - Deleting batches ArrayBuffer(1427871400000
ms)
2015-04-01 08:56:50 DEBUG BlockManagerSlaveActor:63 - Sent response: 0 to Actor[akka://sparkDriver/temp/$o]
2015-04-01 08:56:50 DEBUG SparkDeploySchedulerBackend:50 - [actor] received message ReviveOffers
from Actor[akka://sparkDriver/user/CoarseGrainedScheduler#1727295119]
2015-04-01 08:56:50 DEBUG TaskSchedulerImpl:63 - parentName: , name: TaskSet_0, runningTasks:
0
2015-04-01 08:56:50 DEBUG SparkDeploySchedulerBackend:56 - [actor] handled message (0.499181
ms) ReviveOffers from Actor[akka://sparkDriver/user/CoarseGrainedScheduler#1727295119]
2015-04-01 08:56:50 WARN  TaskSchedulerImpl:71 - Initial job has not accepted any resources;
check your cluster UI to ensure that workers are registered and have sufficient resources
2015-04-01 08:56:51 DEBUG SparkDeploySchedulerBackend:50 - [actor] received message ReviveOffers
from Actor[akka://sparkDriver/user/CoarseGrainedScheduler#1727295119]
2015-04-01 08:56:51 DEBUG TaskSchedulerImpl:63 - parentName: , name: TaskSet_0, runningTasks:
0
2015-04-01 08:56:51 DEBUG SparkDeploySchedulerBackend:56 - [actor] handled message (0.886121
ms) ReviveOffers from Actor[akka://sparkDriver/user/CoarseGrainedScheduler#1727295119]
2015-04-01 08:56:52 DEBUG AppClient$ClientActor:50 - [actor] received message ExecutorUpdated(0,EXITED,Some(Command
exited with code 1),Some(1)) from Actor[akka.tcp://sparkMaster@somesparkhost:7077/user/Master#336117298]
2015-04-01 08:56:52 INFO  AppClient$ClientActor:59 - Executor updated: app-20150401065621-0007/0
is now EXITED (Command exited with code 1)
2015-04-01 08:56:52 INFO  SparkDeploySchedulerBackend:59 - Executor app-20150401065621-0007/0
removed: Command exited with code 1
2015-04-01 08:56:52 DEBUG SparkDeploySchedulerBackend:50 - [actor] received message RemoveExecutor(0,Unknown
executor exit code (1)) from Actor[akka://sparkDriver/temp/$p]
2015-04-01 08:56:52 ERROR SparkDeploySchedulerBackend:75 - Asked to remove non-existent executor
0
2015-04-01 08:56:52 DEBUG SparkDeploySchedulerBackend:56 - [actor] handled message (1.394052
ms) RemoveExecutor(0,Unknown executor exit code (1)) from Actor[akka://sparkDriver/temp/$p]
2015-04-01 08:56:52 DEBUG AppClient$ClientActor:56 - [actor] handled message (6.653705 ms)
ExecutorUpdated(0,EXITED,Some(Command exited with code 1),Some(1)) from Actor[akka.tcp://sparkMaster@somesparkhost:7077/user/Master#336117298]
2015-04-01 08:56:52 DEBUG AppClient$ClientActor:50 - [actor] received message ExecutorAdded(1,worker-20150331133159-somesparkhost-49556,somesparkhost:49556,2,512)
from Actor[akka.tcp://sparkMaster@somesparkhost:7077/user/Master#336117298]
2015-04-01 08:56:52 INFO  AppClient$ClientActor:59 - Executor added: app-20150401065621-0007/1
on worker-20150331133159-somesparkhost-49556 (somesparkhost:49556) with 2 cores
2015-04-01 08:56:52 INFO  SparkDeploySchedulerBackend:59 - Granted executor ID app-20150401065621-0007/1
on hostPort somesparkhost:49556 with 2 cores, 512.0 MB RAM
2015-04-01 08:56:52 DEBUG AppClient$ClientActor:56 - [actor] handled message (1.119731 ms)
ExecutorAdded(1,worker-20150331133159-somesparkhost-49556,somesparkhost:49556,2,512) from
Actor[akka.tcp://sparkMaster@somesparkhost:7077/user/Master#336117298]
2015-04-01 08:56:52 DEBUG AppClient$ClientActor:50 - [actor] received message ExecutorUpdated(1,LOADING,None,None)
from Actor[akka.tcp://sparkMaster@somesparkhost:7077/user/Master#336117298]
2015-04-01 08:56:52 INFO  AppClient$ClientActor:59 - Executor updated: app-20150401065621-0007/1
is now LOADING
2015-04-01 08:56:52 DEBUG AppClient$ClientActor:56 - [actor] handled message (0.516301 ms)
ExecutorUpdated(1,LOADING,None,None) from Actor[akka.tcp://sparkMaster@somesparkhost:7077/user/Master#336117298]
2015-04-01 08:56:52 DEBUG SparkDeploySchedulerBackend:50 - [actor] received message ReviveOffers
from Actor[akka://sparkDriver/user/CoarseGrainedScheduler#1727295119]
2015-04-01 08:56:52 DEBUG TaskSchedulerImpl:63 - parentName: , name: TaskSet_0, runningTasks:
0
2015-04-01 08:56:52 DEBUG SparkDeploySchedulerBackend:56 - [actor] handled message (0.652891
ms) ReviveOffers from Actor[akka://sparkDriver/user/CoarseGrainedScheduler#1727295119]
2015-04-01 08:56:52 DEBUG AppClient$ClientActor:50 - [actor] received message ExecutorUpdated(1,RUNNING,None,None)
from Actor[akka.tcp://sparkMaster@somesparkhost:7077/user/Master#336117298]
2015-04-01 08:56:52 INFO  AppClient$ClientActor:59 - Executor updated: app-20150401065621-0007/1
is now RUNNING
2015-04-01 08:56:52 DEBUG AppClient$ClientActor:56 - [actor] handled message (0.381614 ms)
ExecutorUpdated(1,RUNNING,None,None) from Actor[akka.tcp://sparkMaster@somesparkhost:7077/user/Master#336117298]
2015-04-01 08:56:53 DEBUG SparkDeploySchedulerBackend:50 - [actor] received message ReviveOffers
from Actor[akka://sparkDriver/user/CoarseGrainedScheduler#1727295119]
2015-04-01 08:56:53 DEBUG TaskSchedulerImpl:63 - parentName: , name: TaskSet_0, runningTasks:
0
2015-04-01 08:56:53 DEBUG SparkDeploySchedulerBackend:56 - [actor] handled message (0.417759
ms) ReviveOffers from Actor[akka://sparkDriver/user/CoarseGrainedScheduler#1727295119]
2015-04-01 08:56:54 DEBUG SparkDeploySchedulerBackend:50 - [actor] received message ReviveOffers
from Actor[akka://sparkDriver/user/CoarseGrainedScheduler#1727295119]
2015-04-01 08:56:54 DEBUG TaskSchedulerImpl:63 - parentName: , name: TaskSet_0, runningTasks:
0
2015-04-01 08:56:54 DEBUG SparkDeploySchedulerBackend:56 - [actor] handled message (1.426392
ms) ReviveOffers from Actor[akka://sparkDriver/user/CoarseGrainedScheduler#1727295119]
2015-04-01 08:56:55 DEBUG RecurringTimer:63 - Callback for JobGenerator called at time 1427871415000
2015-04-01 08:56:55 DEBUG JobGenerator:63 - Got event GenerateJobs(1427871415000 ms)
2015-04-01 08:56:55 DEBUG DStreamGraph:63 - Generating jobs for time 1427871415000 ms
2015-04-01 08:56:55 DEBUG KafkaInputDStream:63 - Time 1427871415000 ms is valid
2015-04-01 08:56:55 DEBUG DStreamGraph:63 - Generated 1 jobs for time 1427871415000 ms
2015-04-01 08:56:55 INFO  JobScheduler:59 - Added jobs for time 1427871415000 ms
2015-04-01 08:56:55 DEBUG JobGenerator:63 - Got event DoCheckpoint(1427871415000 ms)
-------------------------------------------
Time: 1427871415000 ms
-------------------------------------------

Mime
View raw message