spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <matei.zaha...@gmail.com>
Subject Re: Bagel?
Date Wed, 14 Aug 2013 23:58:05 GMT
Hmm, it's weird that it built two. It should just be spark-0.7.3/bagel/target.

Matei

On Aug 14, 2013, at 2:29 PM, Ryan Compton <compton.ryan@gmail.com> wrote:

> spark-0.7.3/bagel/target or spark-0.7.3/bagel/bagel/target ?
> 
> On Wed, Aug 7, 2013 at 9:17 PM, Matei Zaharia <matei.zaharia@gmail.com> wrote:
>> Hi Ryan,
>> 
>> To use Bagel, you can just add the spark-bagel.jar built within the bagel/target
directory, in addition to the spark-core-assembly. It doesn't have other dependencies beyond
Spark Core.
>> 
>> GraphX is an upcoming project to do richer types of graph processing, but it's fine
to use Bagel for now. We may eventually update the Bagel API to run on top of GraphX.
>> 
>> Matei
>> 
>> On Aug 7, 2013, at 8:54 PM, Ryan Compton <compton.ryan@gmail.com> wrote:
>> 
>>> I'm interested in processing graphs with Spark. I'm able to do sbt/sbt
>>> assembly and the process exits with success. I can find:
>>> 
>>> -bash-3.2$ find . | grep jar$ | grep -assembly-
>>> ./core/target/spark-core-assembly-0.7.3cdh3u3.jar
>>> ./streaming/target/spark-streaming-assembly-0.7.3cdh3u3.jar
>>> 
>>> I add these to my local Nexus repo, include them in my pom.xml, and
>>> build/run distributed jobs just fine.
>>> 
>>> Now, how do I code up iterative graph algorithms? "sbt/sbt assembly"
>>> didn't make a Bagel assembly.
>>> 
>>> Also, I can't find example code. The code here:
>>> https://github.com/mesos/spark/wiki/Bagel-Programming-Guide has lots
>>> of @serializable depreciation warnings and I can't find the ./run
>>> script they are taking about at the bottom of this page:
>>> http://spark-project.org/docs/latest/bagel-programming-guide.html
>>> 
>>> I get the impression that Bagel may be discontinued. I also noticed
>>> some ads for something called "GraphX" on the BDAS website. Should I
>>> be using GraphX instead?
>>> 
>>> On Wed, Aug 7, 2013 at 7:05 PM, Ryan Compton <compton.ryan@gmail.com> wrote:
>>>> I'm interested in processing graphs with Spark. I'm able to do
>>>> sbt/sbt assembly and the process exits with success. I can find:
>>>> 
>>>> -bash-3.2$ find . | grep jar$ | grep -assembly-
>>>> ./core/target/spark-core-assembly-0.7.3cdh3u3.jar
>>>> ./streaming/target/spark-streaming-assembly-0.7.3cdh3u3.jar
>>>> 
>>>> I add these to my local Nexus repo, include them in my pom.xml, and
>>>> build/run distributed jobs just fine.
>>>> 
>>>> Now, how do I code up iterative graph algorithms? "sbt/sbt assembly"
>>>> didn't make a Bagel assembly.
>>>> 
>>>> Also, I can't find example code. The code here:
>>>> https://github.com/mesos/spark/wiki/Bagel-Programming-Guide has lots
>>>> of @serializable depreciation warnings and I can't find the ./run
>>>> script they are taking about at the bottom of this page:
>>>> http://spark-project.org/docs/latest/bagel-programming-guide.html
>>>> 
>>>> I get the impression that Bagel may be discontinued. Is there
>>>> something else I can use for graphs?
>>>> 
>>>> On Wed, Aug 7, 2013 at 7:02 PM, Ryan Compton <compton.ryan@gmail.com>
wrote:
>>>>> I'm interesting in processing graphs with Spark. I'm able to do
>>>>> sbt/sbt assembly and the process exits with success. I can find:
>>>>> 
>>>>> -bash-3.2$ find . | grep jar$ | grep -assembly-
>>>>> ./core/target/spark-core-assembly-0.7.3cdh3u3.jar
>>>>> ./streaming/target/spark-streaming-assembly-0.7.3cdh3u3.jar
>>>>> 
>>>>> I add these to my local Nexus repo, include them in my pom.xml, and
>>>>> build/run distributed jobs just fine.
>>>>> 
>>>>> Now, how do I code up iterative graph algorithms? "sbt/sbt assembly"
>>>>> didn't make a Bagel assembly.
>>>>> 
>>>>> Also, I can't find example code. The code here:
>>>>> https://github.com/mesos/spark/wiki/Bagel-Programming-Guide has lots
>>>>> of @serializable depreciation warnings and I can't find the ./run
>>>>> script they are taking about at the bottom of this page:
>>>>> http://spark-project.org/docs/latest/bagel-programming-guide.html
>>>>> 
>>>>> I get the impression that Bagel may be discontinued. Is there
>>>>> something else I can use for graphs?
>> 


Mime
View raw message