samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chinmay Soman <chinmay.cere...@gmail.com>
Subject Re: Kafka topic naming conventions
Date Thu, 19 Mar 2015 01:21:55 GMT
Yeah ! It does seem a bit hackish - but I think this approach promises less
config/operation errors.

Although I think some of these checks can be built within Samza - assuming
Kafka has a metadata store in the near future - the Samza container can
validate the #topics against this store.

On Wed, Mar 18, 2015 at 6:16 PM, Chris Riccomini <criccomini@apache.org>
wrote:

> Hey Chinmay,
>
> Cool, this is good feedback. I didn't think I was *that* crazy. :)
>
> Cheers,
> Chris
>
> On Wed, Mar 18, 2015 at 6:10 PM, Chinmay Soman <chinmay.cerebro@gmail.com>
> wrote:
>
> > Thats what we're doing as well - appending partition count to the kafka
> > topic name. This actually helps keep track of the #partitions for each
> > topic (since Kafka doesn't have a Metadata store yet).
> >
> > In case of topic expansion - we actually just resort to creating a new
> > topic. Although that is an overhead - the thought process is that this
> will
> > minimize operational errors. Also, this is necessary to do in case we're
> > doing some kind of joins.
> >
> >
> > On Wed, Mar 18, 2015 at 5:59 PM, Jakob Homan <jghoman@gmail.com> wrote:
> >
> > > On 18 March 2015 at 17:48, Chris Riccomini <criccomini@apache.org>
> > wrote:
> > > > One thing I haven't seen, but might be relevant, is including
> partition
> > > > counts in the topic.
> > >
> > > Yeah, but then if you change the partition count later on, you've got
> > > incorrect information forever. Or you need to create a new stream,
> > > which might be a nice forcing function to make sure your join isn't
> > > screwed up.  There'd need to be something somewhere to enforce that
> > > though.
> > >
> >
> >
> >
> > --
> > Thanks and regards
> >
> > Chinmay Soman
> >
>



-- 
Thanks and regards

Chinmay Soman

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message