spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koert Kuipers <ko...@tresata.com>
Subject Re: how to publish spark inhouse?
Date Fri, 01 Aug 2014 15:38:46 GMT
ok on the positive side i managed to publish inhouse snapshots with maven.

on the negative side with the exclusions in dependencies getting lost when
the pom gets read into sbt i do not think sbt build is currently usable.
well, at least not for building distributions, i tried that and jboss netty
makes it way into the dependencies which causes the fun exception when you
run a spark job:
*org*.*jboss*.*netty*.*channel*.*ChannelException*: *Failed to bind to ...*


On Tue, Jul 29, 2014 at 4:17 PM, Koert Kuipers <koert@tresata.com> wrote:

> SBT does actually have a runtime scrope, but things might indeed not
> translate perfectly.
>
> however exclusions being ignored by the sbt-pom-reader is a serious issue.
> not yet sure how to deal with that one
>
>
> On Tue, Jul 29, 2014 at 4:08 PM, Sean Owen <sowen@cloudera.com> wrote:
>
>> Regarding SBT: I don't think SBT has equivalents for some Maven
>> concepts like "runtime" scope. The resulting assembly-like JARs might
>> not be what's intended. I think that's part of why Maven is the build
>> of reference. You might be able to get the SBT build to do the right
>> thing with fiddling. It was my understanding that sbt-pom-reader
>> didn't quite support everything required to make the same artifact. So
>> I think SBT is not suitable for making release-like things, even for
>> an internal snapshot.
>>
>> deploy is the simplest goal for publishing things like snapshots, yes.
>> It just makes artifacts and pushes them, without going through a lot
>> of release rigmarole like setting up versions, tests, tagging,
>> committing, etc. I use it for simple snapshots.
>>
>> mvn install is the even simpler thing, as it deploys your artifacts to
>> your local machine repo, where you can depend on them locally.
>> Neither of those two update all the versions for you AFAIK. Sorry.
>> It's usually for snapshots, where the version is already set to the
>> right snapshot version by default.
>>
>> Getting to know the Maven plugins takes some work, but eventually you
>> save more time and errors using them than doing it manually.
>>
>> distributionManagement is inherited from the standard Apache parent.
>>
>> On Tue, Jul 29, 2014 at 8:58 PM, Koert Kuipers <koert@tresata.com> wrote:
>> > all i want to do is 1) change the version number and 2) publish spark
>> to our
>> > internal maven repo.
>> >
>> > i just learned that the maven way what i want to do is a "deploy", not a
>> > "release" (neither of which makes sense to me.. how about publish? but
>> ok).
>> > although me wanting to change a version number is apparently too much to
>> > ask, without using a plugin (the maven-release-plugin?). thats fine. i
>> will
>> > ignore that and go change all the files by hand.
>> >
>> > so i now understand i need to do a mvn deploy, and there should be a
>> > distributionManagement section somewhere in the pom. ok great i can try
>> > adding that myself. but how come it isnt there by default? i mean, spark
>> > does end up on mavem somehow?
>> >
>> >
>> >
>> >
>> > On Mon, Jul 28, 2014 at 3:39 PM, Koert Kuipers <koert@tresata.com>
>> wrote:
>> >>
>> >> ah ok thanks. guess i am gonna read up about maven-release-plugin then!
>> >>
>> >>
>> >> On Mon, Jul 28, 2014 at 3:37 PM, Sean Owen <sowen@cloudera.com> wrote:
>> >>>
>> >>> This is not something you edit yourself. The Maven release plugin
>> >>> manages setting all this. I think virtually everything you're worried
>> >>> about is done for you by this plugin.
>> >>>
>> >>> Maven requires artifacts to set a version and it can't inherit one.
I
>> >>> feel like I understood the reason this is necessary at one point.
>> >>>
>> >>> On Mon, Jul 28, 2014 at 8:33 PM, Koert Kuipers <koert@tresata.com>
>> wrote:
>> >>> > and if i want to change the version, it seems i have to change
it in
>> >>> > all 23
>> >>> > pom files? mhhh. is it mandatory for these sub-project pom files
to
>> >>> > repeat
>> >>> > that version info? useful?
>> >>> >
>> >>> > spark$ grep 1.1.0-SNAPSHOT * -r  | wc -l
>> >>> > 23
>> >>> >
>> >>> >
>> >>> >
>> >>> > On Mon, Jul 28, 2014 at 3:05 PM, Koert Kuipers <koert@tresata.com>
>> >>> > wrote:
>> >>> >>
>> >>> >> hey we used to publish spark inhouse by simply overriding the
>> >>> >> publishTo
>> >>> >> setting. but now that we are integrated in SBT with maven i
cannot
>> >>> >> find it
>> >>> >> anymore.
>> >>> >>
>> >>> >> i tried looking into the pom file, but after reading 1144 lines
of
>> xml
>> >>> >> i
>> >>> >> 1) havent found anything that looks like publishing
>> >>> >> 2) i feel somewhat sick too
>> >>> >> 3) i am considering alternative careers to developing...
>> >>> >>
>> >>> >> where am i supposed to look?
>> >>> >> thanks for your help!
>> >>> >
>> >>> >
>> >>
>> >>
>> >
>>
>
>

Mime
View raw message