spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "A.K.M. Ashrafuzzaman" <ashrafuzzaman...@gmail.com>
Subject Re: Connection pool in workers
Date Sun, 01 Mar 2015 12:00:14 GMT
Sorry guys may bad,
Here is a high level code sample,

val unionStreams = ssc.union(kinesisStreams)
unionStreams.foreachRDD(rdd => {
  rdd.foreach(tweet =>
    val strTweet = new String(tweet, "UTF-8")
    val interaction = InteractionParser.parser(strTweet)
    interactionDAL.insert(interaction)
  )
})

Here I have to close the connection for interactionDAL other wise the JVM gives me error that
the connection is open. I tried with sticky connection as well with keep_alive true. So my
guess was that at the point of “unionStreams.foreachRDD” or at “rdd.foreach” the code
is marshaled and send to workers and workers un-marshals and execute the process, which is
why the connection is alway opened for each RDD. I might be completely wrong. I would love
to know what is going on underneath.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message