spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emre Sevinc <emre.sev...@gmail.com>
Subject Re: Why does consuming a RESTful web service (using javax.ws.rs.* and Jsersey) work in unit test but not when submitted to Spark?
Date Wed, 24 Dec 2014 14:44:51 GMT
Sean,

Thanks a lot for the important information, especially  userClassPathFirst.

Cheers,
Emre

On Wed, Dec 24, 2014 at 3:38 PM, Sean Owen <sowen@cloudera.com> wrote:

> That could well be it -- oops, I forgot to run with the YARN profile
> and so didn't see the YARN dependencies. Try the userClassPathFirst
> option to try to make your app's copy take precedence.
>
> The second error is really a JVM bug, but, is from having too little
> memory available for the unit tests.
>
>
> http://spark.apache.org/docs/latest/building-spark.html#setting-up-mavens-memory-usage
>
> On Wed, Dec 24, 2014 at 1:56 PM, Emre Sevinc <emre.sevinc@gmail.com>
> wrote:
> > It seems like YARN depends an older version of Jersey, that is 1.9:
> >
> >   https://github.com/apache/spark/blob/master/yarn/pom.xml
> >
> > When I've modified my dependencies to have only:
> >
> >   <dependency>
> >       <groupId>com.sun.jersey</groupId>
> >       <artifactId>jersey-core</artifactId>
> >       <version>1.9.1</version>
> >     </dependency>
> >
> > And then modified the code to use the old Jersey API:
> >
> >     Client c = Client.create();
> >     WebResource r = c.resource("http://localhost:2222/rest")
> >                      .path("annotate")
> >                      .queryParam("text",
> > UrlEscapers.urlFragmentEscaper().escape(spotlightSubmission))
> >                      .queryParam("confidence", "0.3");
> >
> >     logger.warn("!!! DEBUG !!! target: {}", r.getURI());
> >
> >     String response = r.accept(MediaType.APPLICATION_JSON_TYPE)
> >                        //.header("")
> >                        .get(String.class);
> >
> >     logger.warn("!!! DEBUG !!! Spotlight response: {}", response);
> >
> > It seems to work when I use spark-submit to submit the application that
> > includes this code.
> >
> > Funny thing is, now my relevant unit test does not run, complaining about
> > not having enough memory:
> >
> > Java HotSpot(TM) 64-Bit Server VM warning: INFO:
> > os::commit_memory(0x00000000c4900000, 25165824, 0) failed; error='Cannot
> > allocate memory' (errno=12)
> > #
> > # There is insufficient memory for the Java Runtime Environment to
> continue.
> > # Native memory allocation (mmap) failed to map 25165824 bytes for
> > committing reserved memory.
> >
> > --
> > Emre
> >
> >
> > On Wed, Dec 24, 2014 at 1:46 PM, Sean Owen <sowen@cloudera.com> wrote:
> >>
> >> Your guess is right, that there are two incompatible versions of
> >> Jersey (or really, JAX-RS) in your runtime. Spark doesn't use Jersey,
> >> but its transitive dependencies may, or your transitive dependencies
> >> may.
> >>
> >> I don't see Jersey in Spark's dependency tree except from HBase tests,
> >> which in turn only appear in examples, so that's unlikely to be it.
> >> I'd take a look with 'mvn dependency:tree' on your own code first.
> >> Maybe you are including JavaEE 6 for example?
> >>
> >> On Wed, Dec 24, 2014 at 12:02 PM, Emre Sevinc <emre.sevinc@gmail.com>
> >> wrote:
> >> > Hello,
> >> >
> >> > I have a piece of code that runs inside Spark Streaming and tries to
> get
> >> > some data from a RESTful web service (that runs locally on my
> machine).
> >> > The
> >> > code snippet in question is:
> >> >
> >> >      Client client = ClientBuilder.newClient();
> >> >      WebTarget target = client.target("http://localhost:2222/rest");
> >> >      target = target.path("annotate")
> >> >                  .queryParam("text",
> >> > UrlEscapers.urlFragmentEscaper().escape(spotlightSubmission))
> >> >                  .queryParam("confidence", "0.3");
> >> >
> >> >       logger.warn("!!! DEBUG !!! target: {}",
> >> > target.getUri().toString());
> >> >
> >> >       String response =
> >> >
> >> >
> target.request().accept(MediaType.APPLICATION_JSON_TYPE).get(String.class);
> >> >
> >> >       logger.warn("!!! DEBUG !!! Spotlight response: {}", response);
> >> >
> >> > When run inside a unit test as follows:
> >> >
> >> >      mvn clean test -Dtest=SpotlightTest#testCountWords
> >> >
> >> > it contacts the RESTful web service and retrieves some data as
> expected.
> >> > But
> >> > when the same code is run as part of the application that is submitted
> >> > to
> >> > Spark, using spark-submit script I receive the following error:
> >> >
> >> >       java.lang.NoSuchMethodError:
> >> >
> >> >
> javax.ws.rs.core.MultivaluedMap.addAll(Ljava/lang/Object;[Ljava/lang/Object;)V
> >> >
> >> > I'm using Spark 1.1.0 and for consuming the web service I'm using
> Jersey
> >> > in
> >> > my project's pom.xml:
> >> >
> >> >      <dependency>
> >> >       <groupId>org.glassfish.jersey.containers</groupId>
> >> >       <artifactId>jersey-container-servlet-core</artifactId>
> >> >       <version>2.14</version>
> >> >     </dependency>
> >> >
> >> > So I suspect that when the application is submitted to Spark, somehow
> >> > there's a different JAR in the environment that uses a different
> version
> >> > of
> >> > Jersey / javax.ws.rs.*
> >> >
> >> > Does anybody know which version of Jersey / javax.ws.rs.*  is used in
> >> > the
> >> > Spark environment, or how to solve this conflict?
> >> >
> >> >
> >> > --
> >> > Emre Sevinç
> >> > https://be.linkedin.com/in/emresevinc/
> >> >
> >
> >
> >
> >
> > --
> > Emre Sevinc
>



-- 
Emre Sevinc

Mime
View raw message