spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin Burton <bur...@spinn3r.com>
Subject Re: Debian package for spark?
Date Sat, 08 Nov 2014 23:12:14 GMT
Weird… I’m using a 1.1.0 source tar.gz …

but if it’s fixed in 1.1.1 that’s good.

On Sat, Nov 8, 2014 at 2:08 PM, Mark Hamstra <mark@clearstorydata.com>
wrote:

> The building of the Debian package in Spark works just fine for me -- I
> just did it using a clean check-out of 1.1.1-SNAPSHOT and `mvn -U -Pdeb
> -DskipTests clean package`.  There's likely something else amiss in your
> build.
>
> Actually, that's not quite true.  There is one small problem with the
> Debian packaging that you should be aware of:
>
> https://issues.apache.org/jira/browse/SPARK-3624
> https://github.com/apache/spark/pull/2477#issuecomment-58291272
>
> You should also know there is no such thing as standard Debian packages or
> official debs for Spark, nor is it likely that there ever will be.  What is
> available was never intended as anything more than a convenient hack (or a
> starting point for a custom hack) for Spark developers or users who need a
> way to create a Spark package sufficient to use in a configuration
> management system or something of that nature.  A proper collection of debs
> that divides Spark up into multiple parts, properly reflects inter-package
> dependencies, relocates executables, configuration and libraries to conform
> to the expectations of a larger system, etc. is something that the Apache
> Spark Project does not do, probably won't do, and probably shouldn't do --
> something like that is better handled by the distributors of OSes or larger
> software systems like Apache Bigtop.
>
> On Sat, Nov 8, 2014 at 1:17 PM, Kevin Burton <burton@spinn3r.com> wrote:
>
>> Another note for the official debs.  ‘spark’ is a bad package name
>> because of confusion with the spark programming lang based on ada.
>>
>> There are packages for this already named ‘spark’
>>
>> so I put mine as ‘apache-spark’
>>
>>
>>
>> On Sat, Nov 8, 2014 at 12:21 PM, Kevin Burton <burton@spinn3r.com> wrote:
>>
>>> OK… here’s my version.
>>>
>>> https://github.com/spinn3r/spark-deb
>>>
>>> it’s just two files really.  so if the standard spark packages get fixed
>>> I’ll just switch to them.
>>>
>>> Doesn’t look like there’s an init script and the conf isn’t in /etc …
>>>
>>> On Sat, Nov 8, 2014 at 12:06 PM, Kevin Burton <burton@spinn3r.com>
>>> wrote:
>>>
>>>> looks like it doesn’t work:
>>>>
>>>> > [ERROR] Failed to execute goal org.vafer:jdeb:0.11:jdeb (default) on
>>>> project spark-assembly_2.10: Failed to create debian package
>>>> /Users/burton/Dropbox/projects-macbook-pro-2013-09/spark-1.1.0/assembly/target/spark_1.1.0-${buildNumber}_all.deb:
>>>> Could not create deb package: Control file descriptor keys are invalid
>>>> [Version]. The following keys are mandatory [Package, Version, Section,
>>>> Priority, Architecture, Maintainer, Description]. Please check your
>>>> pom.xml/build.xml and your control file. -> [Help 1]
>>>>
>>>> On Sat, Nov 8, 2014 at 11:24 AM, Kevin Burton <burton@spinn3r.com>
>>>> wrote:
>>>>
>>>>> Nice!  Not sure how I missed that.  Building it now.  If it has all
>>>>> the init scripts and config in the right place I might use that.
>>>>>
>>>>> I might have to build a cassandra package too which adds cassandra
>>>>> support.. I *think* at least.
>>>>>
>>>>> Maybe distribute this .deb with the standard downloads?
>>>>>
>>>>> Kevin
>>>>>
>>>>> On Sat, Nov 8, 2014 at 11:19 AM, Eugen Cepoi <cepoi.eugen@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Yep there is one have a look here
>>>>>> http://spark.apache.org/docs/latest/building-with-maven.html#building-spark-debian-packages
>>>>>> Le 8 nov. 2014 19:48, "Kevin Burton" <burton@spinn3r.com> a
écrit :
>>>>>>
>>>>>> Are there debian packages for spark?
>>>>>>>
>>>>>>> If not I plan on making one… I threw one together in about
20
>>>>>>> minutes as they are somewhat easy with maven and jdeb.  But of
course there
>>>>>>> are other things I need to install like cassandra support and
an init
>>>>>>> script.
>>>>>>>
>>>>>>> So I figured I’d ask here first.
>>>>>>>
>>>>>>> If not we will open source our packaging code and put it on github.
>>>>>>> It’s about 50 lines of code :-P
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Founder/CEO Spinn3r.com
>>>>>>> Location: *San Francisco, CA*
>>>>>>> blog: http://burtonator.wordpress.com
>>>>>>> … or check out my Google+ profile
>>>>>>> <https://plus.google.com/102718274791889610666/posts>
>>>>>>> <http://spinn3r.com>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Founder/CEO Spinn3r.com
>>>>> Location: *San Francisco, CA*
>>>>> blog: http://burtonator.wordpress.com
>>>>> … or check out my Google+ profile
>>>>> <https://plus.google.com/102718274791889610666/posts>
>>>>> <http://spinn3r.com>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Founder/CEO Spinn3r.com
>>>> Location: *San Francisco, CA*
>>>> blog: http://burtonator.wordpress.com
>>>> … or check out my Google+ profile
>>>> <https://plus.google.com/102718274791889610666/posts>
>>>> <http://spinn3r.com>
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Founder/CEO Spinn3r.com
>>> Location: *San Francisco, CA*
>>> blog: http://burtonator.wordpress.com
>>> … or check out my Google+ profile
>>> <https://plus.google.com/102718274791889610666/posts>
>>> <http://spinn3r.com>
>>>
>>>
>>
>>
>> --
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> blog: http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> <https://plus.google.com/102718274791889610666/posts>
>> <http://spinn3r.com>
>>
>>
>


-- 

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>

Mime
View raw message