spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From zzzzzqf12345 <>
Subject Spark Streaming NeteorkReceiver problems
Date Fri, 06 Jun 2014 05:21:53 GMT
here is problem description, I write a custom networkreceiver to receive
image data from camera. I had confirmed all the data received correctly.

1)when data received, only the networkreceiver node run at full speed, while
other nodes keep idle, my spark cluster has 6 nodes.

2)And every image data is calculated many times, which I expected to
calculate once.

3)How to distribute tasks to the whole cluster?

I tried, problems in 2) is still existed ,and
sometimes job faidled.

here is spark configration,
conf.set("spark.executor.memory", "3g");
conf.set("spark.shuffle.netty.connect.timeout", "300000")
conf.set("", "300000")

node : 8 cores, mem 6G.
NetWork : 1Gb/bps

Any suggestions will be appreciated. 


View this message in context:
Sent from the Apache Spark User List mailing list archive at

View raw message