spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emre Sevinc <emre.sev...@gmail.com>
Subject Re: How to use Spark Streaming .jar file that I've built using a different branch than master?
Date Mon, 20 Apr 2015 12:43:25 GMT
Apparently, after *only* building Spark Streaming, I also have to:

   mvn --projects assembly/ -DskipTests clean install

so that my test project uses the new version when I pass it to spark-submit.

--
Emre Sevinç


On Mon, Apr 20, 2015 at 10:58 AM, Emre Sevinc <emre.sevinc@gmail.com> wrote:

> Hello,
>
> I'm building a different version of Spark Streaming (based on a different
> branch than master) in my application for testing purposes, but it seems
> like spark-submit is ignoring my newly built Spark Streaming .jar, and
> using an older version.
>
> Here's some context:
>
> I'm on a different branch:
>
> $ git branch
> * SPARK-3276
>   master
>
> Then I build the Spark Streaming that I've changed:
>
> ✔ ~/code/spark [SPARK-3276 L|✚ 1]
> $ mvn --projects streaming/ -DskipTests install
>
> it builds without problems, and then when I check my local Maven
> repository, I see that I have newly generated Spark Streaming jars:
>
> $ ls -lh
> ~/.m2/repository/org/apache/spark/spark-streaming_2.10/1.4.0-SNAPSHOT/
> total 3.3M
> -rw-rw-r-- 1 emre emre 1.6K Apr 20 10:43 maven-metadata-local.xml
> -rw-rw-r-- 1 emre emre  421 Apr 20 10:43 _remote.repositories
> -rw-rw-r-- 1 emre emre 1.3M Apr 20 10:42
> spark-streaming_2.10-1.4.0-SNAPSHOT.jar
> -rw-rw-r-- 1 emre emre 622K Apr 20 10:43
> spark-streaming_2.10-1.4.0-SNAPSHOT-javadoc.jar
> -rw-rw-r-- 1 emre emre 6.7K Apr 20 10:42
> spark-streaming_2.10-1.4.0-SNAPSHOT.pom
> -rw-rw-r-- 1 emre emre 181K Apr 20 10:42
> spark-streaming_2.10-1.4.0-SNAPSHOT-sources.jar
> -rw-rw-r-- 1 emre emre 1.2M Apr 20 10:42
> spark-streaming_2.10-1.4.0-SNAPSHOT-tests.jar
> -rw-rw-r-- 1 emre emre  82K Apr 20 10:42
> spark-streaming_2.10-1.4.0-SNAPSHOT-test-sources.jar
>
> Then I build and run an application (in Java) that uses Spark Streaming.
> In that test project's pom.xml I have
>
> ...
>  <properties>
>     <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
>     <hadoop.version>2.4.0</hadoop.version>
>     <spark.version>1.4.0-SNAPSHOT</spark.version>
>   </properties>
> ...
>  <dependency>
>       <groupId>org.apache.spark</groupId>
>       <artifactId>spark-streaming_2.10</artifactId>
>       <version>${spark.version}</version>
>       <scope>provided</scope>
>     </dependency>
>
>
> And then I use
>
>   ~/code/spark/bin/spark-submit
>
> to submit my application. It starts fine, and continues to run on my local
> filesystem but when I check the log messages on the console, I don't see
> the changes I have made, and I *did* make changes, e.g. changed some
> logging messages. It is like when I submit my application, it is not using
> the Spark Streaming from *branch SPARK-3276* but from the master branch.
>
> Any ideas what might be causing this? Is there some form of caching? Or is
> spark-submit using a different .jar for streaming? (Where?)
>
> How can I see the effects of my changes that I did to Spark Streaming in
> my SPARK-3276 branch?
>
> --
> Emre Sevinç
>



-- 
Emre Sevinc

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message