spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Hamstra <m...@clearstorydata.com>
Subject Re: Debian package for spark?
Date Sat, 08 Nov 2014 23:45:02 GMT
No change from 1.1.0 to 1.1.1-SNAPSHOT.  The deb profile hasn't changed
since before the 1.0.2 release.

On Sat, Nov 8, 2014 at 3:12 PM, Kevin Burton <burton@spinn3r.com> wrote:

> Weird… I’m using a 1.1.0 source tar.gz …
>
> but if it’s fixed in 1.1.1 that’s good.
>
> On Sat, Nov 8, 2014 at 2:08 PM, Mark Hamstra <mark@clearstorydata.com>
> wrote:
>
>> The building of the Debian package in Spark works just fine for me -- I
>> just did it using a clean check-out of 1.1.1-SNAPSHOT and `mvn -U -Pdeb
>> -DskipTests clean package`.  There's likely something else amiss in your
>> build.
>>
>> Actually, that's not quite true.  There is one small problem with the
>> Debian packaging that you should be aware of:
>>
>> https://issues.apache.org/jira/browse/SPARK-3624
>> https://github.com/apache/spark/pull/2477#issuecomment-58291272
>>
>> You should also know there is no such thing as standard Debian packages
>> or official debs for Spark, nor is it likely that there ever will be.  What
>> is available was never intended as anything more than a convenient hack (or
>> a starting point for a custom hack) for Spark developers or users who need
>> a way to create a Spark package sufficient to use in a configuration
>> management system or something of that nature.  A proper collection of debs
>> that divides Spark up into multiple parts, properly reflects inter-package
>> dependencies, relocates executables, configuration and libraries to conform
>> to the expectations of a larger system, etc. is something that the Apache
>> Spark Project does not do, probably won't do, and probably shouldn't do --
>> something like that is better handled by the distributors of OSes or larger
>> software systems like Apache Bigtop.
>>
>> On Sat, Nov 8, 2014 at 1:17 PM, Kevin Burton <burton@spinn3r.com> wrote:
>>
>>> Another note for the official debs.  ‘spark’ is a bad package name
>>> because of confusion with the spark programming lang based on ada.
>>>
>>> There are packages for this already named ‘spark’
>>>
>>> so I put mine as ‘apache-spark’
>>>
>>>
>>>
>>> On Sat, Nov 8, 2014 at 12:21 PM, Kevin Burton <burton@spinn3r.com>
>>> wrote:
>>>
>>>> OK… here’s my version.
>>>>
>>>> https://github.com/spinn3r/spark-deb
>>>>
>>>> it’s just two files really.  so if the standard spark packages get
>>>> fixed I’ll just switch to them.
>>>>
>>>> Doesn’t look like there’s an init script and the conf isn’t in /etc
…
>>>>
>>>> On Sat, Nov 8, 2014 at 12:06 PM, Kevin Burton <burton@spinn3r.com>
>>>> wrote:
>>>>
>>>>> looks like it doesn’t work:
>>>>>
>>>>> > [ERROR] Failed to execute goal org.vafer:jdeb:0.11:jdeb (default)
on
>>>>> project spark-assembly_2.10: Failed to create debian package
>>>>> /Users/burton/Dropbox/projects-macbook-pro-2013-09/spark-1.1.0/assembly/target/spark_1.1.0-${buildNumber}_all.deb:
>>>>> Could not create deb package: Control file descriptor keys are invalid
>>>>> [Version]. The following keys are mandatory [Package, Version, Section,
>>>>> Priority, Architecture, Maintainer, Description]. Please check your
>>>>> pom.xml/build.xml and your control file. -> [Help 1]
>>>>>
>>>>> On Sat, Nov 8, 2014 at 11:24 AM, Kevin Burton <burton@spinn3r.com>
>>>>> wrote:
>>>>>
>>>>>> Nice!  Not sure how I missed that.  Building it now.  If it has all
>>>>>> the init scripts and config in the right place I might use that.
>>>>>>
>>>>>> I might have to build a cassandra package too which adds cassandra
>>>>>> support.. I *think* at least.
>>>>>>
>>>>>> Maybe distribute this .deb with the standard downloads?
>>>>>>
>>>>>> Kevin
>>>>>>
>>>>>> On Sat, Nov 8, 2014 at 11:19 AM, Eugen Cepoi <cepoi.eugen@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Yep there is one have a look here
>>>>>>> http://spark.apache.org/docs/latest/building-with-maven.html#building-spark-debian-packages
>>>>>>> Le 8 nov. 2014 19:48, "Kevin Burton" <burton@spinn3r.com>
a écrit :
>>>>>>>
>>>>>>> Are there debian packages for spark?
>>>>>>>>
>>>>>>>> If not I plan on making one… I threw one together in about
20
>>>>>>>> minutes as they are somewhat easy with maven and jdeb.  But
of course there
>>>>>>>> are other things I need to install like cassandra support
and an init
>>>>>>>> script.
>>>>>>>>
>>>>>>>> So I figured I’d ask here first.
>>>>>>>>
>>>>>>>> If not we will open source our packaging code and put it
on
>>>>>>>> github.  It’s about 50 lines of code :-P
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Founder/CEO Spinn3r.com
>>>>>>>> Location: *San Francisco, CA*
>>>>>>>> blog: http://burtonator.wordpress.com
>>>>>>>> … or check out my Google+ profile
>>>>>>>> <https://plus.google.com/102718274791889610666/posts>
>>>>>>>> <http://spinn3r.com>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Founder/CEO Spinn3r.com
>>>>>> Location: *San Francisco, CA*
>>>>>> blog: http://burtonator.wordpress.com
>>>>>> … or check out my Google+ profile
>>>>>> <https://plus.google.com/102718274791889610666/posts>
>>>>>> <http://spinn3r.com>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Founder/CEO Spinn3r.com
>>>>> Location: *San Francisco, CA*
>>>>> blog: http://burtonator.wordpress.com
>>>>> … or check out my Google+ profile
>>>>> <https://plus.google.com/102718274791889610666/posts>
>>>>> <http://spinn3r.com>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Founder/CEO Spinn3r.com
>>>> Location: *San Francisco, CA*
>>>> blog: http://burtonator.wordpress.com
>>>> … or check out my Google+ profile
>>>> <https://plus.google.com/102718274791889610666/posts>
>>>> <http://spinn3r.com>
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Founder/CEO Spinn3r.com
>>> Location: *San Francisco, CA*
>>> blog: http://burtonator.wordpress.com
>>> … or check out my Google+ profile
>>> <https://plus.google.com/102718274791889610666/posts>
>>> <http://spinn3r.com>
>>>
>>>
>>
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
>

Mime
View raw message