commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Craig McClanahan <>
Subject Re: [chain] Pipeline implementation
Date Sat, 02 Oct 2004 19:07:04 GMT
On Fri, 17 Sep 2004 11:28:07 -0600, Kris Nuttycombe
<> wrote:
> Hi, all,
> I'm writing to get some advice and perhaps offer some code that may be
> useful to the commons-chain project or elsewhere.


Sorry it took so long (catching up after a two week trip was more time
consuming than I would have dreamed :-).  I've checked in your
proposed "pipeline" code as a separate package in
jakarta-commons-sandbox, and as of tonight you'll also see nightly
builds at:

I had to tweak the Maven dependencies to get commons-digester-1.6.jar
instead of commons-digester-1.6-dev.jar, and I generated an Ant
build.xml file (since my nightly builds use that); other than that,
the code should be exactly as you sent it to me.

I look forward to seeing how this kind of thing can be used, and how
it might interplay with [chain].


PS:  Until Kris is voted as a committer, other sandbox committers
should feel free to help post any patches that are proposed if I'm not
keeping up in a timely manner.

> The group I work for does a large amount of data processing and we are
> working on solutions for pipelined data processing. Our current
> implementation uses a pipeline model where each stage in the pipeline
> has an integrated blocking queue and an abstract process(Object o)
> method that is sequentially applied to each element in the queue. When a
> stage is finished processing, it may pass the processed object (or
> products derived from it) onto the input queue of one or more subsequent
> stages in the pipeline. Branching pipelines are supported, and the whole
> mess is configured using Digester.
> There's a lot of similarity here with the chain of responsibility
> pattern that commons-chain implements, but subtle differences as well.
> Each stage runs in one or more separate threads and we are working to
> allow the processing to be distributed across the network. The pipeline
> model assumes that each object placed in the pipe is going to be
> processed by every stage, whereas to my understanding the chain of
> responsibility is more designed for finding an appropriate command to
> use to process a given context. Also, the pipeline is designed to run as
> a service where data can be provided for processing by automated
> systems. For example, data being beamed down from a satellite can be
> aggregated into orbits that are then passed into the pipeline for
> generation of geolocated gridded products, statistical analysis, etc.
> Our group would really like to be able to contribute some of this code
> back to the commons effort, since we use a ton of commons components.
> The amount of overlap with commons-chain is significant, but I'm not
> sure it's a perfect match because of the differing goals. Does anyone
> out there know of other similar efforts? Is there a place for this sort
> of code in commons? Are we just missing something fundamental about
> commons-chain where we should simply be using that instead?
> Suggestions would be much appreciated. I'm happy to send code, examples,
> and documentation to anyone who's interested.
> Thanks,
> Kris
> --
> =====================================================
> Kris Nuttycombe
> Associate Scientist
> Enterprise Data Systems Group
> CIRES, National Geophysical Data Center/NOAA
> (303) 497-6337
> =====================================================
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message