spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emre Sevinc <>
Subject Why does consuming a RESTful web service (using* and Jsersey) work in unit test but not when submitted to Spark?
Date Wed, 24 Dec 2014 12:02:26 GMT

I have a piece of code that runs inside Spark Streaming and tries to get
some data from a RESTful web service (that runs locally on my machine). The
code snippet in question is:

     Client client = ClientBuilder.newClient();
     WebTarget target ="http://localhost:2222/rest");
     target = target.path("annotate")
                 .queryParam("confidence", "0.3");

      logger.warn("!!! DEBUG !!! target: {}", target.getUri().toString());

      String response =

      logger.warn("!!! DEBUG !!! Spotlight response: {}", response);

When run inside a unit test as follows:

     mvn clean test -Dtest=SpotlightTest#testCountWords

it contacts the RESTful web service and retrieves some data as expected.
But when the same code is run as part of the application that is submitted
to Spark, using spark-submit script I receive the following error:


I'm using Spark 1.1.0 and for consuming the web service I'm using Jersey in
my project's pom.xml:


So I suspect that when the application is submitted to Spark, somehow
there's a different JAR in the environment that uses a different version of
Jersey /*

Does anybody know which version of Jersey /*  is used in the
Spark environment, or how to solve this conflict?

Emre Sevinç

View raw message