spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emre Sevinc <>
Subject How to use Spark Streaming .jar file that I've built using a different branch than master?
Date Mon, 20 Apr 2015 08:58:54 GMT

I'm building a different version of Spark Streaming (based on a different
branch than master) in my application for testing purposes, but it seems
like spark-submit is ignoring my newly built Spark Streaming .jar, and
using an older version.

Here's some context:

I'm on a different branch:

$ git branch
* SPARK-3276

Then I build the Spark Streaming that I've changed:

✔ ~/code/spark [SPARK-3276 L|✚ 1]
$ mvn --projects streaming/ -DskipTests install

it builds without problems, and then when I check my local Maven
repository, I see that I have newly generated Spark Streaming jars:

$ ls -lh
total 3.3M
-rw-rw-r-- 1 emre emre 1.6K Apr 20 10:43 maven-metadata-local.xml
-rw-rw-r-- 1 emre emre  421 Apr 20 10:43 _remote.repositories
-rw-rw-r-- 1 emre emre 1.3M Apr 20 10:42
-rw-rw-r-- 1 emre emre 622K Apr 20 10:43
-rw-rw-r-- 1 emre emre 6.7K Apr 20 10:42
-rw-rw-r-- 1 emre emre 181K Apr 20 10:42
-rw-rw-r-- 1 emre emre 1.2M Apr 20 10:42
-rw-rw-r-- 1 emre emre  82K Apr 20 10:42

Then I build and run an application (in Java) that uses Spark Streaming. In
that test project's pom.xml I have


And then I use


to submit my application. It starts fine, and continues to run on my local
filesystem but when I check the log messages on the console, I don't see
the changes I have made, and I *did* make changes, e.g. changed some
logging messages. It is like when I submit my application, it is not using
the Spark Streaming from *branch SPARK-3276* but from the master branch.

Any ideas what might be causing this? Is there some form of caching? Or is
spark-submit using a different .jar for streaming? (Where?)

How can I see the effects of my changes that I did to Spark Streaming in my
SPARK-3276 branch?

Emre Sevinç

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message