spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tamas Jambor <jambo...@gmail.com>
Subject Re: dynamically change receiver for a spark stream
Date Wed, 21 Jan 2015 11:20:16 GMT
Hi Gerard,

thanks, that makes sense. I'll try that out.

Tamas

On Wed, Jan 21, 2015 at 11:14 AM, Gerard Maas <gerard.maas@gmail.com> wrote:

> Hi Tamas,
>
> I meant not changing the receivers, but starting/stopping the Streaming
> jobs. So you would have a 'small' Streaming job for a subset of streams
> that you'd configure->start->stop  on demand.
> I haven't tried myself yet, but I think it should also be possible to
> create a Streaming Job from the Spark Job Server (
> https://github.com/spark-jobserver/spark-jobserver). Then you would have
> a REST interface that even gives you the possibility of passing a
> configuration.
>
> -kr, Gerard.
>
> On Wed, Jan 21, 2015 at 11:54 AM, Tamas Jambor <jamborta@gmail.com> wrote:
>
>> we were thinking along the same line, that is to fix the number of
>> streams and change the input and output channels dynamically.
>>
>> But could not make it work (seems that the receiver is not allowing any
>> change in the config after it started).
>>
>> thanks,
>>
>> On Wed, Jan 21, 2015 at 10:49 AM, Gerard Maas <gerard.maas@gmail.com>
>> wrote:
>>
>>> One possible workaround could be to orchestrate launch/stopping of
>>> Streaming jobs on demand as long as the number of jobs/streams stay
>>> within the boundaries of the resources (cores) you've available.
>>> e.g. if you're using Mesos, Marathon offers a REST interface to manage
>>> job lifecycle. You will still need to solve the dynamic configuration
>>> through some alternative channel.
>>>
>>> On Wed, Jan 21, 2015 at 11:30 AM, Tamas Jambor <jamborta@gmail.com>
>>> wrote:
>>>
>>>> thanks for the replies.
>>>>
>>>> is this something we can get around? Tried to hack into the code
>>>> without much success.
>>>>
>>>> On Wed, Jan 21, 2015 at 3:15 AM, Shao, Saisai <saisai.shao@intel.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I don't think current Spark Streaming support this feature, all the
>>>>> DStream lineage is fixed after the context is started.
>>>>>
>>>>> Also stopping a stream is not supported, instead currently we need to
>>>>> stop the whole streaming context to meet what you want.
>>>>>
>>>>> Thanks
>>>>> Saisai
>>>>>
>>>>> -----Original Message-----
>>>>> From: jamborta [mailto:jamborta@gmail.com]
>>>>> Sent: Wednesday, January 21, 2015 3:09 AM
>>>>> To: user@spark.apache.org
>>>>> Subject: dynamically change receiver for a spark stream
>>>>>
>>>>> Hi all,
>>>>>
>>>>> we have been trying to setup a stream using a custom receiver that
>>>>> would pick up data from sql databases. we'd like to keep that stream
>>>>> context running and dynamically change the streams on demand, adding
and
>>>>> removing streams based on demand. alternativel, if a stream is fixed,
is it
>>>>> possible to stop a stream, change to config and start again?
>>>>>
>>>>> thanks,
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/dynamically-change-receiver-for-a-spark-stream-tp21268.html
>>>>> Sent from the Apache Spark User List mailing list archive at
>>>>> Nabble.com.
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org For
>>>>> additional commands, e-mail: user-help@spark.apache.org
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message