spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akshay Bhardwaj <akshay.bhardwaj1...@gmail.com>
Subject Spark Elasticsearch Connector | Index and Update
Date Fri, 10 May 2019 06:52:27 GMT
Hi All,

Are there any users who have integrated spark structured streaming  with
elastic search 6.x?

In elastic search doc, ElasticSearch Hadoop Configuration
<https://www.elastic.co/guide/en/elasticsearch/hadoop/6.6/configuration.html>
There is a property es.write.option which is defined as follows

es.write.operation (default index)
The write operation elasticsearch-hadoop should perform - can be any of:


   1. index (default)
   - new data is added while existing data (based on its id) is replaced
         (reindexed).
      1. create
   - adds new data - if the data already exists (based on its id), an
         exception is thrown.
      1. update
   - updates existing data (based on its id). If no data is found, an
         exception is thrown.
      1. upsert
   - known as merge or insert if the data does not exist, updates if the
         data exists (based on its id).


*Say if I have 2 documents in the partition, and based on a field I want to
index the document, and based on another field I want to update the
document with an inline script.*

*Is there a possibility to do this in the same writeStream for Elastic
Search in spark structured streaming?*

Akshay Bhardwaj
+91-97111-33849

Mime
View raw message