camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hart, James W." <>
Subject I'm having trouble processing batches of records from kafka in camel. I can't get them in a batch, nor can I assemble them using aggregation.
Date Thu, 07 Mar 2019 16:43:51 GMT
1. I am trying to process batches of records in camel from kafka, but I am unable to get a
batch, nor am I able to aggregate the records myself.  I was expecting to get 5 records in
a batch because I set "maxPollRecords=5", and then I could process them in a batch or split
and process them individually.  I think that I see that the route did fetch the batch from
kafka as on the 5th record I see that the header kafka.LAST_RECORD_BEFORE_COMMIT is true and
the others prior are false, so I think that confirms that I got 5 records.  Does this indicate
the batch fetch from kafka worked?

2. I also tried to aggregate using an aggregation strategy and end the aggregation on the
kafka.LAST_RECORD_BEFORE_COMMIT=true record , but my aggregation strategy never assembled
the records.  My camel is rusty as after a 5 year gap I'm back doing camel coding, so it may
just be that I'm rusty.

I need to enrich the data, and I am able to enrich at scale by doing batches and running a
bulk query from my database, but I can only do this if I can batch them.

My route defined this way in java:
	.process(new SetHeadersForRouting(partitionSetup))
	.aggregate(header("topic.PARTITION"), new ArrayListAggregationStrategy()).completionPredicate(new

Does camel support this batch consuming from kafka and allow me to process them in a batch?
Should I expect that camel would deliver as a batch, and I just need to tweak my configuration,
or do I need to aggregate and have just not gotten my aggregation code correct?

I am running camel version 2.22.3 and am running in java 1.8 in a fairly new java version.

Any help would be much appreciated!

View raw message