nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ulisses Franca <francaulissesmig...@gmail.com>
Subject [PutElasticSearchProcessor] - Put Elastic search Processor with External version
Date Tue, 06 Nov 2018 23:55:19 GMT
Hi,
We have been using Apache Nifi for a while and its a core part of our
system.
Our main use case is to consume some documents from a apache kafka cluster
and persist it to an Elastic search cluster. The version of these documents
is controlled externally. We use the PutElasticSearch5processor to push
data to elastic search, currently we are interested in the most recent
version of a document, but we insert all the updates in elastic search and
rely on searching to get the latest version of the document.
As you can see this creates a lot of redundant copies of the same document,
which we dont use.
We are exploring options to overcome this problem, and one of them would be
to extend the PutElasticSearch5Processor and add support for external
version of the elastic search documents. We checked and the elastic search
_bulk api supports external versioning.

My question is simple, is there any plans to add support for external
version on the PuElasticSearch5Processor ?

Regards,
Ulisses Franca

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message