flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Flink Elasticsearch Connector: Lucene Error message
Date Fri, 14 Jul 2017 09:15:46 GMT
This kind of error almost always hints at a dependency clash, i.e. there is some version of
this code in the class path that clashed with the version that the Flink program uses. That’s
why it works in local mode, where there are probably not many other dependencies and not in
cluster mode.

How are you running it on the cluster? Standalone, YARN?

Best,
Aljoscha
> On 13. Jul 2017, at 13:56, Fabian Wollert <fabian.wollert@zalando.de> wrote:
> 
> Hi Timo, Hi Gordon,
> 
> thx for the reply! I checked the connection from both clusters to each other, and i can
telnet to the 9300 port of flink, so i think the connection is not an issue here. 
> 
> We are currently using in our live env a custom elasticsearch connector, which used some
extra lib's deployed on the cluster. i found one lucene lib and deleted it (since all dependencies
should be in the flink job jar), but that unfortunately did not help neither ...
> 
> Cheers
> Fabian
> 
> 
> --
> Fabian Wollert
> Data Engineering
> Technology
> 
> E-Mail: fabian.wollert@zalando.de <mailto:fabian.wollert@zalando.de>
> Location: ZMAP <http://zmap.zalando.net/?q=fabian.wollert@zalando.de>
> 2017-07-13 13:46 GMT+02:00 Timo Walther <twalthr@apache.org <mailto:twalthr@apache.org>>:
> Hi Fabian,
> 
> I loop in Gordon. Maybe he knows whats happening here. 
> 
> Regards,
> Timo
> 
> 
> Am 13.07.17 um 13:26 schrieb Fabian Wollert:
>> Hi everyone,
>> 
>> I'm trying to make use of the new Elasticsearch Connector <https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/elasticsearch.html>.
I got a version running locally (with ssh tunnels to my Elasticsearch cluster in AWS) in my
IDE, I see the data in Elasticsearch written perfectly, as I want it. As soon as I try to
run this on our dev cluster (Flink 1.3.0, running in the same VPC like ) though, i get the
following error message (in the sink):
>> 
>> java.lang.NoSuchFieldError: LUCENE_5_5_0
>> 	at org.elasticsearch.Version.<clinit>(Version.java:295)
>> 	at org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:129)
>> 	at org.apache.flink.streaming.connectors.elasticsearch2.Elasticsearch2ApiCallBridge.createClient(Elasticsearch2ApiCallBridge.java:65)
>> 	at org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase.open(ElasticsearchSinkBase.java:272)
>> 	at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
>> 	at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:111)
>> 	at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:375)
>> 	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:252)
>> 	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
>> 	at java.lang.Thread.run(Thread.java:748)
>> 
>> I first thought that this has something to do with mismatched versions, but it happens
to me with Elasticsearch 2.2.2 (bundled with Lucene 5.4.1) and Elasticsearch 2.3 (bundled
with Lucene 5.5.0). 
>> 
>> Can someone point to what exact version conflict is happening here (or where to investigate
further)? Currently my set up looks like everything is actually running with Lucene 5.5.0,
so I'm wondering where that error message is exactly coming from. And also why it is running
locally, but not in the cluster. I'm still investigating if this is a general connection issue
from the Flink cluster to the ES cluster, but that would be surprising, and also that error
message would be then misleading ....
>> 
>> Cheers
>> Fabian
>> 
>> --
>> Fabian Wollert
>> Senior Data Engineer
>> 
>> POSTAL ADDRESS
>> Zalando SE
>> 11501 Berlin
>> 
>> OFFICE
>> Zalando SE
>> Charlottenstraße 4
>> 10969 Berlin
>> Germany
>> 
>> Email: fabian.wollert@zalando.de <mailto:fabian.wollert@zalando.de>
>> Web: corporate.zalando.com <http://corporate.zalando.com/>
>> Jobs: jobs.zalando.de <http://jobs.zalando.de/>
>> 
>> Zalando SE, Tamara-Danz-Straße 1, 10243 Berlin
>> Company registration: Amtsgericht Charlottenburg, HRB 158855 B
>> VAT registration number: DE 260543043
>> Management Board: Robert Gentz, David Schneider, Rubin Ritter
>> Chairperson of the Supervisory Board: Lothar Lanz
>> Registered office: Berlin
> 
> 


Mime
View raw message