flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Pompermaier <pomperma...@okkam.it>
Subject Multiple ElasticSearch sinks
Date Fri, 10 Jul 2015 15:02:56 GMT
Hi to all,

I have a Flink job that produce json objects that I'd like to index in
different Elasticsearch indices depending on the "type" attribute of my
json object (e.g. "people", "places", etc..).
Is there any previous attempt to do something like that in Flink?
I was thinking to use the EsHadoopOutputFormat but it requires to specify
the index name in the job conf..however, in my use case I'll know the
target indices only once the computation finish so Flink can't know how
many sinks there will be in the pre-flight phase..

My solution at the moment was to implement my own mapPartition function
that instantiate a client to ES and index the json documents in the right
index at the end of the job pipeline..is there any better approach to it?

Best,
Flavio

Mime
View raw message