nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Burgess <mattyb...@apache.org>
Subject Re: [PutElasticSearchProcessor] - Put Elastic search Processor with External version
Date Wed, 07 Nov 2018 01:20:19 GMT
Ulisses,

There is a Jira [1] along with a patch [2] to add this feature, but it
appears to have languished for most of the year. I will leave a
comment on the PR to see if the author is interested in continuing the
work; otherwise, perhaps someone else from the community will pick it
up.

Regards,
Matt

[1] https://issues.apache.org/jira/browse/NIFI-4625
[2] https://github.com/apache/nifi/pull/2287
On Tue, Nov 6, 2018 at 6:59 PM Ulisses Franca
<francaulissesmiguel@gmail.com> wrote:
>
> Hi,
> We have been using Apache Nifi for a while and its a core part of our
> system.
> Our main use case is to consume some documents from a apache kafka cluster
> and persist it to an Elastic search cluster. The version of these documents
> is controlled externally. We use the PutElasticSearch5processor to push
> data to elastic search, currently we are interested in the most recent
> version of a document, but we insert all the updates in elastic search and
> rely on searching to get the latest version of the document.
> As you can see this creates a lot of redundant copies of the same document,
> which we dont use.
> We are exploring options to overcome this problem, and one of them would be
> to extend the PutElasticSearch5Processor and add support for external
> version of the elastic search documents. We checked and the elastic search
> _bulk api supports external versioning.
>
> My question is simple, is there any plans to add support for external
> version on the PuElasticSearch5Processor ?
>
> Regards,
> Ulisses Franca

Mime
View raw message