buildr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shane Witbeck" <sh...@digitalsanctum.com>
Subject Re: Transitive dependencies, version matching and OSGi support
Date Thu, 08 May 2008 13:15:38 GMT
This looks like I can move to using environments instead of my pre-1.3
custom extension. Thanks for the pointer.

On Thu, May 8, 2008 at 12:03 AM, Assaf Arkin <arkin@intalio.com> wrote:
> On Wed, May 7, 2008 at 7:44 PM, Shane Witbeck <shane@digitalsanctum.com>
> wrote:
>
>> Expanding on the notion of deploying, I extended Buildr to allow
>> deployments to various server environments (local, development,
>> production, etc.). The command looks something like this:
>>
>> buildr projectname:deploy-dev
>>
>> The above command automagically packages and deploys the artifact(s)
>> to a specific server which makes it nice for turning around changes
>> quickly. To accomplish this I used a combination of extending
>> Buildr::Project and the Net::SFTP project.
>
>
> Have you looked into environments in 1.3?  You could do something like:
>
> buildr -e dev projectname:deploy
>
> or set the environment variable BUILDR_ENV=dev.
>
> http://incubator.apache.org/buildr/settings_profiles.html#environments
>
> Assaf
>
>
>>
>>
>>
>>
>>
>> On Wed, May 7, 2008 at 9:44 PM, Assaf Arkin <arkin@intalio.com> wrote:
>> > On Wed, May 7, 2008 at 6:19 PM, Shane Witbeck <shane@digitalsanctum.com>
>> >  wrote:
>> >
>> >
>> >  > I second the idea of going the way of OSGI. It's clearly the next step
>> >  > in Java's evolution.
>> >  >
>> >  > In terms of pre-1.3 dependency support, we resorted to doing something
>> >  > like the following to support a subset of compile dependencies for
>> >  > runtime packages:
>> >  >
>> >  > project.runtime.with(specs)
>> >
>> >
>> >  +1 for adding runtime dependencies.
>> >
>> >
>> >
>> >  >
>> >  >
>> >  > Because some of our environments had shared runtime classpaths, it
>> >  > wasn't always necessary to include certain libraries. I did this by
>> >  > extending Buildr::Project to support the different kinds of internal
>> >  > projects we have such as WebProject, StandaloneProject, BatchProject,
>> >  > etc.
>> >  >
>> >  > Something else that was needed in our projects was a way of putting
>> >  > together different packages for different environments. Again, by
>> >  > extending Buildr::Project, I introduced a way to package and deploy
>> >  > (via SSH) the various projects.
>> >  >
>> >  > I mention the ways that I've extended Buildr to possibly touch on new
>> >  > features to incorporate such as:
>> >  >
>> >  > 1. Being able to define different sets of dependencies for projects
>> >  > (in my case it was compile vs. runtime). I like the direction Assaf
>> >  > mentions with being able to merge dependency arrays and better
>> >  > transitive support. This is huge in terms of being able to make the
>> >  > buildfile less verbose.
>> >  > 2. A way to easily deploy (via SSH, etc.) packages
>> >
>> >
>> >  Can you expand on that?
>> >
>> >
>> >
>> >  >
>> >  > 3. Better integration with IDE's. It's a struggle to keep the
>> >  > dependencies in sync without having to regenerate the IDE files and in
>> >  > the process losing other project-specific IDE settings. Perhaps this
>> >  > is better solved via IDE-specific plugins for Buildr?
>> >
>> >
>> >  Good point.  It might be possible to retain those project-specific IDE
>> >  settings, though I don't know what it would look like.
>> >
>> >
>> >
>> >  >
>> >  > 4. In terms of additional language support, my vote is for Flex. With
>> >  > the recent open sourced SDK, I believe there will be a surge in usage
>> >  > among not only Java developers but developers of other languages as
>> >  > well.
>> >
>> >
>> >  We'll need someone who's using Flex to help develop it.
>> >
>> >  Also, this is something we don't have to tie to the release schedule.
>>  We
>> >  finally have an API for adding new languages and delivering extensions
>> using
>> >  Ruby Gems, so you can get people to use it without waiting for the next
>> >  release.  Separately, once we add it to the code base it will show up in
>> the
>> >  next release, but I think independent Gem would be the way to start.
>> >
>> >  Assaf
>> >
>> >
>> >
>> >
>> >  >
>> >  >
>> >  > I'd be glad to elaborate further on the points I've brought up and
>> >  > look forward to contributing more.
>> >  >
>> >  > Thanks,
>> >  > Shane
>> >  >
>> >  >
>> >  > On Fri, May 2, 2008 at 5:41 PM, Assaf Arkin <arkin@intalio.com>
>> wrote:
>> >  > > Now that 1.3 is out, it's time to start talking about future
>> releases.
>> >  > >   We've got the 1.3.x line of releases, for dealing with outstanding
>> >  > issues,
>> >  > >  new ones we'll find along the way, and building on new features
>> that
>> >  > showed
>> >  > >  up in 1.3.
>> >  > >
>> >  > >  Separately, there's 1.4, the next major release and a place to make
>> more
>> >  > >  significant changes.  We just have to decide what those would be.
>> >  > >
>> >  > >  I propose that we focus 1.4 on improving the way we handle
>> dependencies.
>> >  > >
>> >  > >  Specifically what I have in mind are transitive dependencies,
>> version
>> >  > >  matching, and OSGi support.  So I'll start by talking about some
>> things
>> >  > I've
>> >  > >  been looking into recently.
>> >  > >
>> >  > >
>> >  > >  For starters, adding a dependency method to artifact (which also
>> >  > includes
>> >  > >  packages) that returns an array of all the known dependencies for
>> that
>> >  > >  artifacts.  Ruby arrays are pretty powerful, you can do things
>> like:
>> >  > >
>> >  > >  # Does it have a particular dependency?
>> >  > >  foo.dependencies.any? { |dep| ... }
>> >  > >  # Add new dependencies
>> >  > >  bar.dependencies << ....
>> >  > >  # Merge foo's dependencies into bar
>> >  > >  bar.dependencies |= foo.dependencies
>> >  > >
>> >  > >  One problem that comes up frequently are broken dependency lists,
>> they
>> >  > may
>> >  > >  be missing a dependency, or reference the wrong version, or include
>> one
>> >  > you
>> >  > >  don't want to use.  Mutable arrays would make it easy to fix these
>> >  > broken
>> >  > >  dependency lists, for example:
>> >  > >
>> >  > >  # Change the artifact in memory
>> >  > >  artifact( spec ).exclude( dont-want-this ).include( but want this
)
>> >  > >  # And use it
>> >  > >  compile.with spec
>> >  > >
>> >  > >  It will be even better to introduce a Dependencies class that
>> extends
>> >  > array
>> >  > >  and adds all sorts of convenience methods and lazy expansion,
>> similar to
>> >  > >  Rake's FileList.  Then you could do things like:
>> >  > >
>> >  > >  # Find dependency by its specification
>> >  > >  foo.dependencies.has?( spec )
>> >  > >  # Exclude all dependencies that match a pattern
>> >  > >  foo.dependencies.exclude!( pattern )
>> >  > >  # Expand all transitive dependencies and return flat list
>> >  > >  foo.dependencies.expand
>> >  > >  # Replace one with another
>> >  > >  foo.dependencies.replace!( dont-want, want )
>> >  > >
>> >  > >  The API for methods like compile.with and compile.dependencies
>> remain
>> >  > the
>> >  > >  same, just changing the underlying implementation.
>> >  > >
>> >  > >  Packages would acquire dependencies from the project, for example,
>> >  > >  package(:jar) would add all the dependencies from
>> compile.dependencies,
>> >  > >  which of course you can fix to your heart's content.
>> >  > >
>> >  > >
>> >  > >  Version matching makes it possible to point at the most recent
>> version
>> >  > that
>> >  > >  satisfies a particular version requirement, so >=1.2 to get 1.2.0
>> or
>> >  > later,
>> >  > >  or ~>1.2.3 to get any version between 1.2.3 and 1.3 (excluding).
>> >  > >
>> >  > >
>> >  > >  OSGi is a set of big specs that boil a lot of oceans, the part I'm
>> >  > >  particularly interested in, is managing dependencies.  When we
>> started
>> >  > >  working on Buildr, OSGi was used almost exclusively by Eclipse.
>>  That
>> >  > >  changed.  We're seeing more and more servers and frameworks
>> adopting
>> >  > OSGi.
>> >  > >
>> >  > >  Imagine that we had OSGi support.  You will specify all the
>> dependencies
>> >  > >  once in the buildfile.  Those will be used to build the project,
>> and
>> >  > also
>> >  > >  incorporated into the package manifest.  Other projects could use
>> these
>> >  > >  packages, reading the dependency list from the package manifest.
>>  More
>> >  > >  important, when you get to deploy those packages, the runtime can
>> use
>> >  > that
>> >  > >  information to bring in the right dependencies.
>> >  > >
>> >  > >  That's one of those things that, once you use it, you won't want
to
>> go
>> >  > back.
>> >  > >
>> >  > >
>> >  > >  The problem is, OSGi and Maven are incompatible with each other.
>>  I'll
>> >  > try
>> >  > >  to summarize the differences:
>> >  > >
>> >  > >  1. The OSGi package identifier is typically the same as the file
>> name.
>> >  >  The
>> >  > >  Maven package identifier combines group and artifact identifiers,
>> only
>> >  > the
>> >  > >  later is used as the file name.
>> >  > >
>> >  > >  2.  Both use four-part version numbers.  The first three are
>> numerical
>> >  > (e.g.
>> >  > >  1.2.3) and managed the same way, the fourth one is the qualifier.
>>  OSGi
>> >  > >  qualifiers are alphanumeric and compared lexically, so R1 comes
>> before
>> >  > R2;
>> >  > >  using number is not advised, 10 comes before 2.  Maven qualifiers
>> are
>> >  > >  numeric for releases, and alphanumeric for pre-releases; the
>> qualifier
>> >  > 10
>> >  > >  comes after 2, and 2 comes after R2.
>> >  > >
>> >  > >  3.  The rules for version matching are therefore not the same. 
The
>> same
>> >  > >  rule could match different versions, depending on whethr you use
>> OSGi or
>> >  > >  Maven.
>> >  > >
>> >  > >  There's also the issue that OSGi keeps the meta-data inside the
>> package,
>> >  > >  while Maven requires the additional POM file, and if both are
>> present we
>> >  > >  have to decide which one to rely on.
>> >  > >
>> >  > >  It's possible to some extent to support both at the same time, but
>> there
>> >  > >  will always be that impedence mismatch.  If we do a good enough
>> job, it
>> >  > will
>> >  > >  probably only affect edge cases, but we need to be aware.
>> >  > >
>> >  > >  It will definitely be easier to pick one of the two, either OSGi
or
>> >  > Maven,
>> >  > >  and add transitive support and version matching based on that
>> model,
>> >  > then do
>> >  > >  translation when necessary (e.g. when reading POM file, generating
>> OSGi
>> >  > >  manifest).
>> >  > >
>> >  > >  If I had to start all over again, I would go for OSGi with Maven
>> >  > >  compatibility layer for three simple reasons:
>> >  > >
>> >  > >  1. Runtime deployment
>> >  > >  2. Simpler to implement, and if your build doesn't quite work,
>> >  > troubleshoot
>> >  > >  and fix.
>> >  > >  3. Works the same way as most other packaging schemes (e.g. Gems,
>> >  > Debian,
>> >  > >  RPM).
>> >  > >
>> >  > >
>> >  > >  Comments, ideas?
>> >  > >
>> >  > >  Assaf
>> >  > >
>> >  >
>> >  >
>> >  >
>> >  > --
>> >  > -Shane
>> >  >
>> >
>>
>>
>>
>> --
>> -Shane
>>
>



-- 
-Shane

Mime
View raw message