flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-9885) End-to-end test: Elasticsearch 6.x connector
Date Mon, 30 Jul 2018 08:25:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-9885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16561627#comment-16561627
] 

ASF GitHub Bot commented on FLINK-9885:
---------------------------------------

twalthr commented on a change in pull request #6391: [FLINK-9885] [FLINK-8101] Finalize Elasticsearch
6.x
URL: https://github.com/apache/flink/pull/6391#discussion_r206042028
 
 

 ##########
 File path: docs/dev/connectors/elasticsearch.md
 ##########
 @@ -151,22 +151,47 @@ List<HttpHost> httpHost = new ArrayList<>();
 httpHosts.add(new HttpHost("127.0.0.1", 9200, "http"));
 httpHosts.add(new HttpHost("10.2.3.1", 9200, "http"));
 
-input.addSink(new ElasticsearchSink<>(httpHosts, new ElasticsearchSinkFunction<String>()
{
-    public IndexRequest createIndexRequest(String element) {
-        Map<String, String> json = new HashMap<>();
-        json.put("data", element);
-    
-        return Requests.indexRequest()
-                .index("my-index")
-                .type("my-type")
-                .source(json);
-    }
-    
-    @Override
-    public void process(String element, RuntimeContext ctx, RequestIndexer indexer) {
-        indexer.add(createIndexRequest(element));
+Map<String, String> bulkRequestsConfig = new HashMap<>();
+// This instructs the sink to emit after every element, otherwise they would be buffered
+bulkRequestsConfig.put("bulk.flush.max.actions", "1");
 
 Review comment:
   We could also think about adding this to the builder now. This is Flink configuration and
we don't need a string map anymore, right?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> End-to-end test: Elasticsearch 6.x connector
> --------------------------------------------
>
>                 Key: FLINK-9885
>                 URL: https://issues.apache.org/jira/browse/FLINK-9885
>             Project: Flink
>          Issue Type: Sub-task
>          Components: ElasticSearch Connector, Tests
>            Reporter: Tzu-Li (Gordon) Tai
>            Assignee: Tzu-Li (Gordon) Tai
>            Priority: Blocker
>              Labels: pull-request-available
>             Fix For: 1.6.0
>
>
> We have decided to try and merge the pending Elasticsearch 6.x PRs. This should also
come with an end-to-end test that covers this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message