spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emre Sevinc <emre.sev...@gmail.com>
Subject Re: Why does consuming a RESTful web service (using javax.ws.rs.* and Jsersey) work in unit test but not when submitted to Spark?
Date Wed, 24 Dec 2014 13:56:03 GMT
It seems like YARN depends an older version of Jersey, that is 1.9:

  https://github.com/apache/spark/blob/master/yarn/pom.xml

When I've modified my dependencies to have only:

  <dependency>
      <groupId>com.sun.jersey</groupId>
      <artifactId>jersey-core</artifactId>
      <version>1.9.1</version>
    </dependency>

And then modified the code to use the old Jersey API:

    Client c = Client.create();
    WebResource r = c.resource("http://localhost:2222/rest")
                     .path("annotate")
                     .queryParam("text",
UrlEscapers.urlFragmentEscaper().escape(spotlightSubmission))
                     .queryParam("confidence", "0.3");

    logger.warn("!!! DEBUG !!! target: {}", r.getURI());

    String response = r.accept(MediaType.APPLICATION_JSON_TYPE)
                       //.header("")
                       .get(String.class);

    logger.warn("!!! DEBUG !!! Spotlight response: {}", response);

It seems to work when I use spark-submit to submit the application that
includes this code.

Funny thing is, now my relevant unit test does not run, complaining about
not having enough memory:

Java HotSpot(TM) 64-Bit Server VM warning: INFO:
os::commit_memory(0x00000000c4900000, 25165824, 0) failed; error='Cannot
allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 25165824 bytes for
committing reserved memory.

--
Emre


On Wed, Dec 24, 2014 at 1:46 PM, Sean Owen <sowen@cloudera.com> wrote:

> Your guess is right, that there are two incompatible versions of
> Jersey (or really, JAX-RS) in your runtime. Spark doesn't use Jersey,
> but its transitive dependencies may, or your transitive dependencies
> may.
>
> I don't see Jersey in Spark's dependency tree except from HBase tests,
> which in turn only appear in examples, so that's unlikely to be it.
> I'd take a look with 'mvn dependency:tree' on your own code first.
> Maybe you are including JavaEE 6 for example?
>
> On Wed, Dec 24, 2014 at 12:02 PM, Emre Sevinc <emre.sevinc@gmail.com>
> wrote:
> > Hello,
> >
> > I have a piece of code that runs inside Spark Streaming and tries to get
> > some data from a RESTful web service (that runs locally on my machine).
> The
> > code snippet in question is:
> >
> >      Client client = ClientBuilder.newClient();
> >      WebTarget target = client.target("http://localhost:2222/rest");
> >      target = target.path("annotate")
> >                  .queryParam("text",
> > UrlEscapers.urlFragmentEscaper().escape(spotlightSubmission))
> >                  .queryParam("confidence", "0.3");
> >
> >       logger.warn("!!! DEBUG !!! target: {}",
> target.getUri().toString());
> >
> >       String response =
> >
> target.request().accept(MediaType.APPLICATION_JSON_TYPE).get(String.class);
> >
> >       logger.warn("!!! DEBUG !!! Spotlight response: {}", response);
> >
> > When run inside a unit test as follows:
> >
> >      mvn clean test -Dtest=SpotlightTest#testCountWords
> >
> > it contacts the RESTful web service and retrieves some data as expected.
> But
> > when the same code is run as part of the application that is submitted to
> > Spark, using spark-submit script I receive the following error:
> >
> >       java.lang.NoSuchMethodError:
> >
> javax.ws.rs.core.MultivaluedMap.addAll(Ljava/lang/Object;[Ljava/lang/Object;)V
> >
> > I'm using Spark 1.1.0 and for consuming the web service I'm using Jersey
> in
> > my project's pom.xml:
> >
> >      <dependency>
> >       <groupId>org.glassfish.jersey.containers</groupId>
> >       <artifactId>jersey-container-servlet-core</artifactId>
> >       <version>2.14</version>
> >     </dependency>
> >
> > So I suspect that when the application is submitted to Spark, somehow
> > there's a different JAR in the environment that uses a different version
> of
> > Jersey / javax.ws.rs.*
> >
> > Does anybody know which version of Jersey / javax.ws.rs.*  is used in the
> > Spark environment, or how to solve this conflict?
> >
> >
> > --
> > Emre Sevinç
> > https://be.linkedin.com/in/emresevinc/
> >
>



-- 
Emre Sevinc

Mime
View raw message