spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From drarse <drarse.a...@gmail.com>
Subject Re: How to keep a SQLContext instance alive in a spark streaming application's life cycle?
Date Tue, 09 Jun 2015 09:41:42 GMT
Why? I  tried  this solution and works fine.

El martes, 9 de junio de 2015, codingforfun [via Apache Spark User List] <
ml-node+s1001560n23218h76@n3.nabble.com> escribió:

> Hi drarse, thanks for replying, the way you said use a singleton object
> does not work
>
>
>
>
> 在 2015-06-09 16:24:25,"drarse [via Apache Spark User List]" <[hidden
> email] <http:///user/SendEmail.jtp?type=node&node=23218&i=0>> 写道:
>
> The best way is create a singleton object like:
>
> object SQLContextSingleton {
>>   @transient private var instance: SQLContext = null
>>
>>   // Instantiate SQLContext on demand
>>   def getInstance(sparkContext: SparkContext): SQLContext = synchronized {
>>     if (instance == null) {
>>       instance = new SQLContext(sparkContext)
>>     }
>>     instance
>>   }}
>>
>>  You have more information in the programming guide:
>
> https://spark.apache.org/docs/latest/streaming-programming-guide.html#dataframe-and-sql-operations
>
>
>
> 2015-06-09 9:27 GMT+02:00 codingforfun [via Apache Spark User List] <[hidden
> email] <http:///user/SendEmail.jtp?type=node&node=23216&i=0>>:
>
>> I used SQLContext in a spark streaming application as blew:
>>
>> ----------------------------------------------------------------------------------------------------------------
>>
>> case class topic_name (f1: Int, f2: Int)
>>
>> val sqlContext = new SQLContext(sc)
>> @transient val ssc = new StreamingContext(sc, new Duration(5 * 1000))
>> ssc.checkpoint(".")
>> val theDStream = KafkaUtils.createDirectStream[String, String,
>> StringDecoder, StringDecoder](ssc, kafkaParams, Set("topic_name"))
>>
>> theDStream.map(x => x._2).foreach { rdd =>
>>   sqlContext.jsonRDD(newsIdRDD).registerTempTable("topic_name")
>>   sqlContext.sql("select count(*) from topic_name").foreach { x =>
>>     WriteToFile("file_path", x(0).toString)
>>   }
>> }
>>
>> ssc.start()
>> ssc.awaitTermination()
>> ----------------------------------------------------------------------------------------------------------------
>>
>>
>> I found i could only get every 5 seconds's count of message, because "The
>> lifetime of this temporary table is tied to the SQLContext that was used to
>> create this DataFrame", i guess every 5 seconds, a new sqlContext will be
>> create and the temporary table can only alive just 5 seconds, i want to the
>> sqlContext and the temporary table alive all the streaming application's
>> life cycle, how to do it?
>>
>> Thanks~
>>
>> ------------------------------
>>  If you reply to this email, your message will be added to the
>> discussion below:
>>
>> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-keep-a-SQLContext-instance-alive-in-a-spark-streaming-application-s-life-cycle-tp23215.html
>>  To start a new topic under Apache Spark User List, email [hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=23216&i=1>
>> To unsubscribe from Apache Spark User List, click here.
>> NAML
>> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-keep-a-SQLContext-instance-alive-in-a-spark-streaming-application-s-life-cycle-tp23215p23216.html
>  To unsubscribe from How to keep a SQLContext instance alive in a spark
> streaming application's life cycle?, click here.
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
>
>
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-keep-a-SQLContext-instance-alive-in-a-spark-streaming-application-s-life-cycle-tp23215p23218.html
>  To start a new topic under Apache Spark User List, email
> ml-node+s1001560n1h49@n3.nabble.com
> <javascript:_e(%7B%7D,'cvml','ml-node%2Bs1001560n1h49@n3.nabble.com');>
> To unsubscribe from Apache Spark User List, click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=ZHJhcnNlLmFtZXNAZ21haWwuY29tfDF8MTUyMzY0MjQyMA==>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>


-- 
Atte. Sergio Jiménez




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-keep-a-SQLContext-instance-alive-in-a-spark-streaming-application-s-life-cycle-tp23215p23219.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Mime
View raw message