kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pere Urbón Bayes <pere.ur...@gmail.com>
Subject Re: Kafka connect FieldPartitioner with scheduled rotation
Date Sun, 03 Feb 2019 17:49:41 GMT
Hi Manu,
  can you share your s3 connector config as well the exception you are
getting? with only this info, I do need more details to understand your
issue. Keep in mind the option you are using "rotate.schedule.interval.ms"
from the docs says:

> This configuration is useful when you have to commit your data based on
current server time, for example at the beginning of every hour. The
default value -1 means that this feature is disabled.

Cheers

Missatge de Manu Jacob <Manu.Jacob@sas.com> del dia dg., 3 de febr. 2019 a
les 16:47:

> Hi,
>
> I want to use s3 connect by portioning with FieldPartitioner and
> partition.field.name set to a non timestamp based field. I want to commit
> and flush based on  both size and time. I am getting an exception when I
> use the option "rotate.schedule.interval.ms". Is it possible to rotate it
> with FieldPartitioner? I tried to set the  timestamp.extractor (with record
> and wallclock) but it looks like it is honored only for time based
> partitioner.
>
> -Manu
>
>
>

-- 
Pere Urbon-Bayes
Software Architect
http://www.purbon.com
https://twitter.com/purbon
https://www.linkedin.com/in/purbon/

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message