kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: Kafka Streams question
Date Tue, 14 Nov 2017 17:42:29 GMT
Boris,

I just realized, that you want to update the state from your processor
-- this is actually not supported by a global state (at least not directly).

Global state is populated from a topic at startup, and the global thread
should be the only thread that updates the state: even if it is
technically possible to write to the global state from any processor,
those updates won't be reflected in the underlying topic and thus might
get lost. Therefore, if you want to update the state, you will need to
directly write to the topic that feeds the state. And than the global
thread will pick those changes up and update the state.

Concurrent access is not save on writes for this reason (we assume
single writer) and you should only read from the global state from your
"regular" processors.


-Matthias

On 11/13/17 1:11 PM, Boris Lublinsky wrote:
> I was thinking about controlled stream use case, where one stream is data for processing,
while the second one controls execution.
> If I want to scale this, I want to run multiple instances. In this case I want these
instances to share data topic, but control topic should be delivered to all
> Instances.
> This means that I would like to control group IDs for streams individually
> 
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com
> https://www.lightbend.com/
> 
>> On Nov 13, 2017, at 2:47 PM, Guozhang Wang <wangguoz@gmail.com> wrote:
>>
>> Boris,
>>
>> What's your use case scenarios that you'd prefer to set different
>> subscriber IDs for different streams?
>>
>>
>> Guozhang
>>
>>
>> On Mon, Nov 13, 2017 at 6:49 AM, Boris Lublinsky <
>> boris.lublinsky@lightbend.com> wrote:
>>
>>> This seems like a very limiting implementation
>>>
>>>
>>> Boris Lublinsky
>>> FDP Architect
>>> boris.lublinsky@lightbend.com
>>> https://www.lightbend.com/
>>>
>>>> On Nov 13, 2017, at 4:21 AM, Damian Guy <damian.guy@gmail.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> The configurations apply to all streams consumed within the same streams
>>>> application. There is no way of overriding it per input stream.
>>>>
>>>> Thanks,
>>>> Damian
>>>>
>>>> On Mon, 13 Nov 2017 at 04:49 Boris Lublinsky <
>>> boris.lublinsky@lightbend.com>
>>>> wrote:
>>>>
>>>>> I am writing Kafka Streams implementation (1.0.0), for which I have 2
>>>>> input streams.
>>>>> Is it possible to have different subscriber IDs for these 2 streams.
>>>>> I see only one place where subscriber’s ID can be specified:
>>>>>
>>>>> streamsConfiguration.put(StreamsConfig.CLIENT_ID_CONFIG,
>>>>> ApplicationKafkaParameters.DATA_GROUP);
>>>>> And it does not seem like either Topology or DSL APIs allow to overwrite
>>>>> it during Stream creation.
>>>>>
>>>>> Thanks for the help
>>>>>
>>>>> Boris Lublinsky
>>>>> FDP Architect
>>>>> boris.lublinsky@lightbend.com
>>>>> https://www.lightbend.com/
>>>>>
>>>>>
>>>
>>>
>>
>>
>> -- 
>> -- Guozhang
> 
> 


Mime
View raw message