flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 周思华 <summerle...@163.com>
Subject Re: Why assignTimestampsAndWatermarks parallelism same as map,it will not fired?
Date Thu, 26 Apr 2018 02:50:19 GMT
Hi 潘,
could you please check the number of kafka's partitions, I think if the {{number of kafka
partition}} <  {{parallelism of source node}}) then there can be some idle parallel which
won't recevice any data...

Best Regards,
Sihua Zhou

On 04/26/2018 10:44,TechnoMage<mlatta@technomage.com> wrote:
If you are using keyed messages in Kafka, or keyed streams in flink, then only partitions
that get hashed to the proper value will get data.  If not keyed messages, then yes they should
all get data.


On Apr 25, 2018, at 8:25 PM, 潘 功森 <pangongsen@hotmail.com> wrote:

The event is running all the time in order,I don't know why one of the partitions does not
receive data if not change parallelism?

发件人: Fabian Hueske <fhueske@gmail.com>
发送时间: 2018年4月25日 10:56
收件人: Timo Walther
抄送: user
主题: Re: Why assignTimestampsAndWatermarks parallelism same as map,it will not fired?

This sounds like one of the partitions does not receive data. Watermark generation is data
driven, i.e., the watermark can only advance if the TimestampAndWatermarkAssigner sees events.

By changing the parallelism between the map and the assigner, the events are shuffled across
and hence there is no "empty" partition anymore.

I would check if one instance of your sources does not emit events.

Best, Fabian

2018-04-25 10:43 GMT+02:00 Timo Walther <twalthr@apache.org>:

did you set your time characteristics to even-time?



Am 25.04.18 um 05:15 schrieb 潘 功森:

Hi all,

I use the same parallelism between map and assignTimestampsAndWatermarks , and it not fired,
I saw the extractTimestamp and generateWatermark all is fine, but watermark is always not
change and keep as min long value.

And then I changed parallelism and different with map, and windows fired.

I used Flink 1.3.2.

Is it a Flink bug?or others can give me why it not fired. It troubled me the whole day.

Best regards,


View raw message