kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "M. Manna" <manme...@gmail.com>
Subject Re: Kafka Deplopyment Using Kubernetes (on Cloud) - settings for log.dirs
Date Mon, 22 Oct 2018 18:22:05 GMT
Yes I have. But it lacks detailed explanation - even the document which I
have obtained through a webinar.

Log.dirs impact, imho, is the most important and relevant item to elaborate
for such publications. And I mean, by using diagrams and details that
apache Kafka and Confluent have done so far (and they’re very good). I’m
sure it’ll be done soon :)

We can give a lot of names e.g. SatefulSets, but I was looking for an
detailed explanation for how this will be different in K8s.


Thanks,

On Mon, 22 Oct 2018 at 19:12, Mark Anderson <manderson23@gmail.com> wrote:

> Have you reviewed
> https://www.confluent.io/blog/getting-started-apache-kafka-kubernetes/ as
> a
> starting point?
>
> On Mon, 22 Oct 2018, 18:07 M. Manna, <manmedia@gmail.com> wrote:
>
> > Thanks a lot for your prompt answer. This is what I was expecting.
> >
> > So, if we had three pods where volumes are mapped as the following
> >
> > Pod 1 = (log.dirs=/some/directory1)
> > Pod 2 = (log.dirs=/some/directory2)
> > Pod 3 = (log.dirs=/some/directory3)
> >
> > if something bad happens to Pod 3 and goes down, would we be able to spin
> > up a new pod with the same logs.dir location?
> >
> > Sorry if I am thinking of something impossible, but kafka and zk are
> > essentially databases and has some stateful items. So just trying to
> > understand how it all becomes different when deployed with K8s on public
> > cloud.
> >
> > Thanks,
> >
> > On Mon, 22 Oct 2018 at 17:55, Svante Karlsson <svante.karlsson@csi.se>
> > wrote:
> >
> > > Different directories, they cannot share path. A broker will delete
> > > everything under the log directory that it does not know about
> > >
> > > Den mån 22 okt. 2018 kl 17:47 skrev M. Manna <manmedia@gmail.com>:
> > >
> > > > Hello,
> > > >
> > > > We are thinking of rolling out Kafka on Kubernetes deployed on public
> > > cloud
> > > > (AWS or GCP, or other). We were hoping if someone could provide some
> > > > suggestion or insight here.
> > > >
> > > > What we are trying to understand is how logs.dir property is affected
> > > when
> > > > we run Pods in a specific worker node? if we are running 3 broker
> PODs
> > > with
> > > > 1 ZK Pod, how are we treating the logs.dir property for brokers (i.e.
> > > their
> > > > location for offsets/data etc.)? Are the log.dirs mounted to
> different
> > > > locations? Or, are they sharing the same path i.e. same set of files?
> > > >
> > > > We understand that people are quite busy with the ongoing release. So
> > any
> > > > insight you can provide will be highly appreciated.
> > > >
> > > > Thanks,
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message