kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: Controlled topics creation in Kafka cluster, how does that affect a Kafka Streams App that uses join?
Date Thu, 06 Sep 2018 00:13:16 GMT
You can figure out the names via


The naming pattern is

You will also need to create those topic with the correct number of
partitions (usually the number of input topic partitions).

With this information, you can pre-create the topics and Kafka Streams
will use them if the exists. Kafka Streams will also check the number of
partitions -- if it's different to what is required, it will fail and
tell you in the exception message how many partitions it expects.


On 9/5/18 8:39 AM, John Roesler wrote:
> Hi Meeiling,
> It sounds like "formal process" is more like "fill out a form beforehand"
> than just "access controlled".
> In that case, although it's somewhat of an internal detail, I think the
> easiest thing would be to figure out what the name of the internal topics
> will be and request their creation before you run the app.
> The names of the internal topics is predictable, based on the application
> id and the app topology, and it will be stable as long as you don't change
> the app.
> I think the simplest approach would be to run it locally and just see which
> topics get created.
> Does that help?
> -John
> On Tue, Sep 4, 2018 at 10:07 PM Meeiling.Bradley <
> Meeiling.Bradley@target.com> wrote:
>> Would I be able to run a Kafka streams application contains join
>> KStream-KStream that consumes and produces to a Kafka cluster that has a
>> formal process to provision topics creation? Since there are behind scene
>> creation of <KSTREAM_JOINOTHER> and <KSTREAM-JOINTHIS> topics need to
>> created for join operation. These topics are created as kafka streams app
>> executes, if topics creation is a controlled process, is there a way to
>> bypass that while my application is running?
>> Meeiling Bradley  |  Lead BI Engineer  |  • EDABI  |   33 South 6th
>> Street, CC-2002  |  Minneapolis, MN 55402

View raw message