kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: Controlled topics creation in Kafka cluster, how does that affect a Kafka Streams App that uses join?
Date Thu, 06 Sep 2018 00:13:16 GMT
You can figure out the names via

builder.build().describe()

The naming pattern is
<application.id>-<operatorName>-[repartition|changelog]

You will also need to create those topic with the correct number of
partitions (usually the number of input topic partitions).

With this information, you can pre-create the topics and Kafka Streams
will use them if the exists. Kafka Streams will also check the number of
partitions -- if it's different to what is required, it will fail and
tell you in the exception message how many partitions it expects.


-Matthias

On 9/5/18 8:39 AM, John Roesler wrote:
> Hi Meeiling,
> 
> It sounds like "formal process" is more like "fill out a form beforehand"
> than just "access controlled".
> 
> In that case, although it's somewhat of an internal detail, I think the
> easiest thing would be to figure out what the name of the internal topics
> will be and request their creation before you run the app.
> 
> The names of the internal topics is predictable, based on the application
> id and the app topology, and it will be stable as long as you don't change
> the app.
> 
> I think the simplest approach would be to run it locally and just see which
> topics get created.
> 
> Does that help?
> -John
> 
> On Tue, Sep 4, 2018 at 10:07 PM Meeiling.Bradley <
> Meeiling.Bradley@target.com> wrote:
> 
>> Would I be able to run a Kafka streams application contains join
>> KStream-KStream that consumes and produces to a Kafka cluster that has a
>> formal process to provision topics creation? Since there are behind scene
>> creation of <KSTREAM_JOINOTHER> and <KSTREAM-JOINTHIS> topics need to
be
>> created for join operation. These topics are created as kafka streams app
>> executes, if topics creation is a controlled process, is there a way to
>> bypass that while my application is running?
>>
>> Meeiling Bradley  |  Lead BI Engineer  |  • EDABI  |   33 South 6th
>> Street, CC-2002  |  Minneapolis, MN 55402
>>
>>
>>
>>
> 


Mime
View raw message