kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis.gospodne...@gmail.com>
Subject Re: Logstash to Kafka
Date Thu, 05 Feb 2015 19:29:44 GMT
Hi,

In short, I don't see Kafka having problems with those numbers.  Logstash
will have a harder time, I believe.
That said, it all depends on how you tune things an what kind of / how much
hardware you use.

2B or 200B events, yes, big numbers, but how quickly do you need to process
those? in 1 minute, 1 hour, 1 day, or a week? :)

SPM for Kafka (http://sematext.com/spm) will show you all possible Kafka
metrics you can imagine, so if you decide to give Kafka a try you'll be
able to tune Kafka with the help of SPM for Kafka charts and the help of
people on this mailing list.

Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/


On Thu, Feb 5, 2015 at 2:12 PM, Vineet Mishra <clearmidoubt@gmail.com>
wrote:

> Yury,
>
> Well thanks for sharing the insight of kafka partition distribution.
>
> Well I am more of a concerned about the throughtput that kafka-storm can
> collaborative give so as to event process.
>
> Currently I am having around a 30 Gb file with around .2 Billion events,
> this number is soon gonna rise 100 times the existing numbers.
>
> I was wondering will the above mentioned stream processing engine will be
> good fit in my case?
> If yes, then with what configuration and tuning so as to effectively use
> resource and maximize throughput.
>
> Thanks!
> On Feb 3, 2015 8:38 PM, "Yury Ruchin" <yuri.ruchin@gmail.com> wrote:
>
> > This is a quote from Kafka documentation:
> > "The routing decision is influenced by the kafka.producer.Partitioner.
> >
> > interface Partitioner<T> {
> >    int partition(T key, int numPartitions);
> > }
> > The partition API uses the key and the number of available broker
> > partitions to return a partition id. This id is used as an index into a
> > sorted list of broker_ids and partitions to pick a broker partition for
> the
> > producer request. The default partitioning strategy is
> > hash(key)%numPartitions. If the key is null, then a random broker
> partition
> > is picked. A custom partitioning strategy can also be plugged in using
> the
> > partitioner.class config parameter."
> >
> > An important point for the null key is that the randomly chosen broker
> > partition sticks for the time specified by "
> > topic.metadata.refresh.interval.ms" which is 10 minutes by default. So
> if
> > you are using null key for Logstash entries, you will be writing to the
> > same partition for 10 minutes. Is this your case?
> >
> > 2015-02-03 14:03 GMT+03:00 Vineet Mishra <clearmidoubt@gmail.com>:
> >
> > > Hi,
> > >
> > > I am having a setup where I am sniffing some logs(ofcourse the big
> ones)
> > > through Logstash Forwarder and forwarding it to Logstash, which in turn
> > > publish these events to Kafka.
> > >
> > > I have created the Kafka Topic ensuring the required number of
> Partitions
> > > and Replication Factor but not sure with Logstash Output
> Configuration, I
> > > am having following doubt with reference to the same.
> > >
> > > For the Logstash Publishing events to kafka
> > >
> > > 1) Do we need to explicitly define the Partition in Logstash while
> > > Publishing to Kafka
> > > 2) Will Kafka take care of the proper distribution of the data across
> the
> > > Partitions
> > >
> > > I am having a notion that despite of the fact of declaring the
> partitions
> > > in Kafka while creating Topic the data from Logstash is been pushed to
> > > single Partition or perhaps not getting uniformly distributed.
> > >
> > > Looking for the Expert Advise.
> > >
> > > Thanks!
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message