kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 杰 杨 <funk...@live.com>
Subject 回复:答复: which Kafka StateStore could I use ?
Date Fri, 02 Mar 2018 19:27:59 GMT
can you show some tips for this?

---原始邮件---
发件人: "Guozhang Wang "<wangguoz@gmail.com>
发送时间: 2018年3月3日 01:32:55
收件人: "users"<users@kafka.apache.org>;
主题: Re: 答复: which Kafka StateStore could I use ?


Hello Jie,

By default Kafka Streams uses caching on top of its internal state stores
to de-dup output streams to the final destination (in your case the DB) so
that for a single key, fewer updates will be generated giving a small
working set. If your aggregation logic follows such key distribution, you
can try enlarge the cache size (by default it is only 50MB) and see if it
helps reduce the downstream traffic to your DB.


Guozhang


On Thu, Mar 1, 2018 at 6:33 PM, 杰 杨 <funkyyj@live.com> wrote:

> Yes .but the DB’s Concurrent quantity is  the limitation.
> Now I can process 600 records/second
> And I want enhance it
>
> 发送自 Windows 10 版邮件<https://go.microsoft.com/fwlink/?LinkId=550986>应用
>
> 发件人: Guozhang Wang<mailto:wangguoz@gmail.com>
> 发送时间: 2018年3月2日 2:59
> 收件人: users@kafka.apache.org<mailto:users@kafka.apache.org>
> 主题: Re: which Kafka StateStore could I use ?
>
> Hello Jie,
>
> Just to understand your problem better, are you referring "db" for an
> external storage engine outside Kafka Streams, and you are asking how to
> only send one record per aggregation key (assuming you are doing some
> aggregations with Streams&apos; statestore) to that end storage engine?
>
>
> Guozhang
>
>
> On Wed, Feb 28, 2018 at 7:53 PM, 杰 杨 <funkyyj@live.com> wrote:
>
> >
> > HI:
> > I use kafka streams for real-time data analysis
> > and I meet a problem.
> > now I process a record in kafka and compute it and send to db.
> > but db concurrency level is not suit for me.
> > so I want that
> > 1)when there is not data in kakfa ,the statestore is  no results.
> > 2) when there is a lot of data records in kafka the statestore save
> > computed result and I need send its once to db.
> > which StateStoe can I use for do that above
> > ________________________________
> > funkyyj@live.com
> >
>
>
>
> --
> -- Guozhang
>
>


--
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message