spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul <corley.paul1...@gmail.com>
Subject Re: is it ok to have multiple sparksession's in one spark structured streaming app?
Date Sat, 09 Sep 2017 03:11:57 GMT
You would set the Kafka topic as your data source and you would write a custom output to Cassandra
everything would be or could be contained within your stream 

-Paul

Sent from my iPhone

> On Sep 8, 2017, at 2:52 PM, kant kodali <kanth909@gmail.com> wrote:
> 
> How can I use one SparkSession to talk to both Kafka and Cassandra let's say?
> 
> 
>> On Fri, Sep 8, 2017 at 3:46 AM, Arkadiusz Bicz <arkadiusz.bicz@gmail.com> wrote:
>> You don't need multiple spark sessions to have more than one stream working, but
from maintenance and reliability perspective it is not good idea. 
>> 
>>> On Thu, Sep 7, 2017 at 2:40 AM, kant kodali <kanth909@gmail.com> wrote:
>>> Hi All,
>>> 
>>> I am wondering if it is ok to have multiple sparksession's in one spark structured
streaming app? Basically, I want to create 1) Spark session for reading from Kafka and 2)
Another Spark session for storing the mutations of a dataframe/dataset to a persistent table
as I get the mutations from #1? 
>>> 
>>> Finally, is this a common practice?
>>> 
>>> Thanks,
>>> kant
>> 
> 

Mime
View raw message