kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Colin <co...@clark.ws>
Subject Re: If you run Kafka in AWS or Docker, how do you persist data?
Date Thu, 05 Mar 2015 05:12:23 GMT
Hello,

We use docker for kafka on vm's with both nas and local disk.  We mount the volumes externally.
 We havent had many problems at all, and a restart has cleared any issue.  We are on .8.1

We are also started to deploy to aws.

--
Colin 
+1 612 859 6129
Skype colin.p.clark

> On Mar 4, 2015, at 10:46 PM, Otis Gospodnetic <otis.gospodnetic@gmail.com> wrote:
> 
> Hi,
> 
>> On Fri, Feb 27, 2015 at 1:36 AM, James Cheng <jcheng@tivo.com> wrote:
>> 
>> Hi,
>> 
>> I know that Netflix might be talking about "Kafka on AWS" at the March
>> meetup, but I wanted to bring up the topic anyway.
>> 
>> I'm sure that some people are running Kafka in AWS.
> 
> 
> I'd say most, not some :)
> 
> 
>> Is anyone running Kafka within docker in production? How does that work?
> 
> Not us.  When I was at DevOps Days in NYC last year, everyone was talking
> about Docker, but only about 2.5 people in the room actually really used it.
> 
> For both of these, how do you persist data? If on AWS, do you use EBS? Do
>> you use ephemeral storage and then rely on replication? And if using
>> docker, do you persist data outside the docker container and on the host
>> machine?
> 
> We've used both EBD and local disks in AWS.  We don't have Kafka
> replication, as far as I know.
> 
> And related, how do you deal with broker failure? Do you simply replace it,
>> and repopulate a new broker via replication? Or do you bring back up the
>> broker with the persisted files?
> 
> We monitor all Kafka pieces - producers, consumer, and brokers with SPM.
> We have alerts and anomaly detection enabled for various Kafka metrics
> (yeah, consumer lag being one of them).
> Broker failures have been very rare (we've used 0.7.2, 0.8.1.x, and are now
> on 0.8.2).  When they happened a restart was typically enough. I can recall
> one instance where segments recovery tool a long time (minutes, maybe more
> than an hour), but this was >6 months ago.
> 
> 
>> Trying to learn about what people are doing, beyond "on premises and
>> dedicated hardware".
> 
> In my world almost everyone I talk to is in AWS.
> 
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message