samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roger Hoover <roger.hoo...@gmail.com>
Subject Re: Kafka topic naming conventions
Date Thu, 19 Mar 2015 04:43:16 GMT
Thanks, guys.  I was also playing around with including partition count and
even the partition key in the topic name.   My thought was that topics may
have the same data and number of partitions but only differ by partition
key.  After a while, the naming does get crazy (too long and ugly).  We
really need a topic metatdata store.

On Wed, Mar 18, 2015 at 6:21 PM, Chinmay Soman <chinmay.cerebro@gmail.com>
wrote:

> Yeah ! It does seem a bit hackish - but I think this approach promises less
> config/operation errors.
>
> Although I think some of these checks can be built within Samza - assuming
> Kafka has a metadata store in the near future - the Samza container can
> validate the #topics against this store.
>
> On Wed, Mar 18, 2015 at 6:16 PM, Chris Riccomini <criccomini@apache.org>
> wrote:
>
> > Hey Chinmay,
> >
> > Cool, this is good feedback. I didn't think I was *that* crazy. :)
> >
> > Cheers,
> > Chris
> >
> > On Wed, Mar 18, 2015 at 6:10 PM, Chinmay Soman <
> chinmay.cerebro@gmail.com>
> > wrote:
> >
> > > Thats what we're doing as well - appending partition count to the kafka
> > > topic name. This actually helps keep track of the #partitions for each
> > > topic (since Kafka doesn't have a Metadata store yet).
> > >
> > > In case of topic expansion - we actually just resort to creating a new
> > > topic. Although that is an overhead - the thought process is that this
> > will
> > > minimize operational errors. Also, this is necessary to do in case
> we're
> > > doing some kind of joins.
> > >
> > >
> > > On Wed, Mar 18, 2015 at 5:59 PM, Jakob Homan <jghoman@gmail.com>
> wrote:
> > >
> > > > On 18 March 2015 at 17:48, Chris Riccomini <criccomini@apache.org>
> > > wrote:
> > > > > One thing I haven't seen, but might be relevant, is including
> > partition
> > > > > counts in the topic.
> > > >
> > > > Yeah, but then if you change the partition count later on, you've got
> > > > incorrect information forever. Or you need to create a new stream,
> > > > which might be a nice forcing function to make sure your join isn't
> > > > screwed up.  There'd need to be something somewhere to enforce that
> > > > though.
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks and regards
> > >
> > > Chinmay Soman
> > >
> >
>
>
>
> --
> Thanks and regards
>
> Chinmay Soman
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message