spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akshay Bhardwaj <>
Subject Spark Elasticsearch Connector | Index and Update
Date Fri, 10 May 2019 06:52:27 GMT
Hi All,

Are there any users who have integrated spark structured streaming  with
elastic search 6.x?

In elastic search doc, ElasticSearch Hadoop Configuration
There is a property es.write.option which is defined as follows

es.write.operation (default index)
The write operation elasticsearch-hadoop should perform - can be any of:

   1. index (default)
   - new data is added while existing data (based on its id) is replaced
      1. create
   - adds new data - if the data already exists (based on its id), an
         exception is thrown.
      1. update
   - updates existing data (based on its id). If no data is found, an
         exception is thrown.
      1. upsert
   - known as merge or insert if the data does not exist, updates if the
         data exists (based on its id).

*Say if I have 2 documents in the partition, and based on a field I want to
index the document, and based on another field I want to update the
document with an inline script.*

*Is there a possibility to do this in the same writeStream for Elastic
Search in spark structured streaming?*

Akshay Bhardwaj

View raw message