kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boris Lublinsky <boris.lublin...@lightbend.com>
Subject Re: Kafka Streams question
Date Tue, 14 Nov 2017 17:46:49 GMT
Its not a global state.
I am using a custom state store

Boris Lublinsky
FDP Architect
boris.lublinsky@lightbend.com
https://www.lightbend.com/

> On Nov 14, 2017, at 11:42 AM, Matthias J. Sax <matthias@confluent.io> wrote:
> 
> Boris,
> 
> I just realized, that you want to update the state from your processor
> -- this is actually not supported by a global state (at least not directly).
> 
> Global state is populated from a topic at startup, and the global thread
> should be the only thread that updates the state: even if it is
> technically possible to write to the global state from any processor,
> those updates won't be reflected in the underlying topic and thus might
> get lost. Therefore, if you want to update the state, you will need to
> directly write to the topic that feeds the state. And than the global
> thread will pick those changes up and update the state.
> 
> Concurrent access is not save on writes for this reason (we assume
> single writer) and you should only read from the global state from your
> "regular" processors.
> 
> 
> -Matthias
> 
> On 11/13/17 1:11 PM, Boris Lublinsky wrote:
>> I was thinking about controlled stream use case, where one stream is data for processing,
while the second one controls execution.
>> If I want to scale this, I want to run multiple instances. In this case I want these
instances to share data topic, but control topic should be delivered to all
>> Instances.
>> This means that I would like to control group IDs for streams individually
>> 
>> Boris Lublinsky
>> FDP Architect
>> boris.lublinsky@lightbend.com
>> https://www.lightbend.com/
>> 
>>> On Nov 13, 2017, at 2:47 PM, Guozhang Wang <wangguoz@gmail.com> wrote:
>>> 
>>> Boris,
>>> 
>>> What's your use case scenarios that you'd prefer to set different
>>> subscriber IDs for different streams?
>>> 
>>> 
>>> Guozhang
>>> 
>>> 
>>> On Mon, Nov 13, 2017 at 6:49 AM, Boris Lublinsky <
>>> boris.lublinsky@lightbend.com> wrote:
>>> 
>>>> This seems like a very limiting implementation
>>>> 
>>>> 
>>>> Boris Lublinsky
>>>> FDP Architect
>>>> boris.lublinsky@lightbend.com
>>>> https://www.lightbend.com/
>>>> 
>>>>> On Nov 13, 2017, at 4:21 AM, Damian Guy <damian.guy@gmail.com>
wrote:
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> The configurations apply to all streams consumed within the same streams
>>>>> application. There is no way of overriding it per input stream.
>>>>> 
>>>>> Thanks,
>>>>> Damian
>>>>> 
>>>>> On Mon, 13 Nov 2017 at 04:49 Boris Lublinsky <
>>>> boris.lublinsky@lightbend.com>
>>>>> wrote:
>>>>> 
>>>>>> I am writing Kafka Streams implementation (1.0.0), for which I have
2
>>>>>> input streams.
>>>>>> Is it possible to have different subscriber IDs for these 2 streams.
>>>>>> I see only one place where subscriber’s ID can be specified:
>>>>>> 
>>>>>> streamsConfiguration.put(StreamsConfig.CLIENT_ID_CONFIG,
>>>>>> ApplicationKafkaParameters.DATA_GROUP);
>>>>>> And it does not seem like either Topology or DSL APIs allow to overwrite
>>>>>> it during Stream creation.
>>>>>> 
>>>>>> Thanks for the help
>>>>>> 
>>>>>> Boris Lublinsky
>>>>>> FDP Architect
>>>>>> boris.lublinsky@lightbend.com
>>>>>> https://www.lightbend.com/
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> -- 
>>> -- Guozhang
>> 
>> 
> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message