commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Sicker <boa...@gmail.com>
Subject Re: Graph status?
Date Mon, 09 Sep 2019 16:10:10 GMT
We have scattered graph logic in Jenkins Core as well as the Pipeline
plugins. If I wanted to use a graph library in Jenkins, it would be
best to use minimal dependencies because any dependencies included
with Jenkins tend to get reused by plugins which can cause upgrade
problems later, especially with libraries that use a lot of
dependencies (particularly dependencies that don't have as strict of
API/ABI compatibility guarantees as typical Commons components).

I bring this up because refactoring the graph APIs in Jenkins are a
loosely defined goal of mine in order to allow for job dependency
graphs to be queried and potentially manipulated more easily. As you
can see here [1], we already have dependency hell going on, so in the
future, I'd like to avoid introducing more unnecessary or hard to
upgrade dependencies (like Guava or anything from Google it seems;
"'we'll just increment the major version for every release and call it
a day" is not a backward compatibility policy).

[1]: https://mvnrepository.com/artifact/org.jenkins-ci.main/jenkins-core/2.194

On Mon, 9 Sep 2019 at 04:40, Romain Manni-Bucau <rmannibucau@gmail.com> wrote:
>
> Le lun. 9 sept. 2019 à 11:06, Rob Vesse <rvesse@dotnetrdf.org> a écrit :
>
> > Playing Devil's advocate:
> >
> > I am always curious when folks complain about a "huge" dependency stack
> > (for a start the term huge is inherently subjective).  This is pretty much
> > the reality of the modern OSS ecosystem, people (yourself included) try to
> > avoid reinventing the wheel and want to focus on solving their problem
> > rather than all the mundane engineering that goes into enabling that.
> >
> > While there are legitimate use cases where keeping the dependency stack
> > minimal is desirable (e.g. embedded computing) for most use cases the size
> > of your dependency stack isn't a major issue.  And if it is there are tools
> > that can aggressively optimise this e.g. ProGuard [1] that are in common
> > usage (e.g. the Android build chain incorporates ProGuard by default)
> >
> > Could you expand on why you see this as being such a problem?
> >
>
> Yep, there are multiple aspects converging to this requirement:
>
> 1. My whole server is 10M + my app specific stack is 3M so adding 10M for a
> graph navigation support would be a killer for the consistency of the app
> (and not it is not what OSS is, spring does that, not the whole apache
> projects ;)
> 2. More your stack is big, harder it is to follow up with CVE and
> dependencies upgrades, therefore for pure algorithm/logic it is saner to
> not depend on anything (and some bad habits like relocations breaks tools
> which can be a disaster if exposed through a server),
> 3. In terms of sharing, any element of a stack should at least be
> understood in a team, here again, more you have, slower you will be or you
> will miss things.
> 4. This logic part will be something highly transferred in clusters so
> smaller it is, better it is (and yes transfered as binaries, not only
> instances).
> 5. Dependency conflicts can be a mess (not here, but since I'm listing the
> general approach I'm mentionning it)
>
> I can hear some people don't care of these points but I have to take care
> of them to ensure I keep a good velocity and control over time of my
> deliveries with a good quality (accross security, responsiveness etc) so
> I'm really concerned about it.
>
>
> >
> > There's clearly a trade off between size of dependency stack and how much
> > code you want to personally develop and maintain but my personal preference
> > is usually to minimize the latter.
> >
>
> I'm in between personally.
> ("What does it cost me to integrate it?" + "What does it cost me for my
> stack?") vs ("What does it cost me to do it now?" + "What does it cost me
> to maintain it?")
>
> Here the cost to do it is low and I evaluate that being dep free and not
> super generic (handling all graph kinds) would be the acceptable compromise
> for the projects I'm targetting (I ack they are maybe specific).
>
>
>
> >
> > Rob
> >
> > [1] https://www.guardsquare.com/en/products/proguard
> >
> > On 08/09/2019, 09:16, "Marco Neumann" <marco.neumann@gmail.com> wrote:
> >
> >     Romain,
> >
> >     sure Jena gives you so much more than just the basic graph
> > infrastructure.
> >     I wasn't acutely aware of the guava shade mandatory requirement in a
> >     minimal viable setup of Jena. Still I would encourage you to engage
> > with
> >     Jena community to discuss design ideas and opportunity for reuse of
> > jena
> >     components in your work on graph at commons and apache.
> >
> >     FYI I currently need the following libraries in a minimal viable setup
> > to
> >     work with the Jena graph api.
> >
> >     jena-base (215kb), jena-core (1.69mb), jena-shaded-guave (2.73mb),
> > log4j
> >     (479kb), slf4j-log (41kb), slf4j-api (12kb)
> >
> >     Marco
> >
> >     On Sat, Sep 7, 2019 at 10:33 PM Romain Manni-Bucau <
> > rmannibucau@gmail.com>
> >     wrote:
> >
> >     > Hi Marco,
> >     >
> >     > Had a look to jena for another project and didnt evaluate it here
> > for these
> >     > reasons (happy to be wrong):
> >     >
> >     > - dep stack was huge for only graph part (guava shade, some other
> > uneeded
> >     > commons etc, most being excludable but without guarantees in time)
> >     > - it is not about DAG and therefore misses navigation methods (which
> > is
> >     > what I need in addition to "mutation" methods for the algo i want to
> > impl)
> >     > - it is not the goal of jena so API and core stack can evolve in an
> >     > undesired manner
> >     >
> >     > To mention alternatives, spark, flink, beam, ignite for the few I
> > can think
> >     > about, have something not crazy but still this stack and API issues
> > :(.
> >     >
> >     > This is how i ended up looking commons, to try to have something
> > stable and
> >     > dep free.
> >     >
> >     > Romain
> >     >
> >     >
> >     > Le sam. 7 sept. 2019 à 23:15, Marco Neumann <marco.neumann@gmail.com>
> > a
> >     > écrit :
> >     >
> >     > > I highly recommend to take a look at the Apache Jena project for
> >     > > inspiration here. It has a very mature graph representationat this
> > point:
> >     > >
> >     > > https://jena.apache.org/
> >     > >
> >     > >
> >     > >
> >     >
> > https://jena.apache.org/documentation/javadoc/jena/org/apache/jena/graph/Graph.html
> >     > >
> >     > > Jena use triples in the form of <v><e><v> to encode
the graph<v,e>
> >     > >
> >     > > give it try and make sure to post to users@jena.apache.org if you
> > have
> >     > any
> >     > > questions
> >     > >
> >     > > enjoy,
> >     > > Marco
> >     > >
> >     > > On Sat, Sep 7, 2019 at 10:30 AM Romain Manni-Bucau <
> >     > rmannibucau@gmail.com>
> >     > > wrote:
> >     > >
> >     > > > Hi all
> >     > > >
> >     > > > What is the status of graph at commons - or apache if we have
> > something
> >     > > > elsewhere?
> >     > > >
> >     > > > I found in sandbox that doc
> >     > > >
> >     > > >
> >     > >
> >     >
> > https://commons.apache.org/sandbox/commons-graph/apidocs/org/apache/commons/graph/DirectedGraph.html
> >     > > > ,
> >     > > > but wonder if we have something live and if not why it failed.
> >     > > >
> >     > > > My rational is I started to write some DAG modelization and
> > tooling
> >     > > > (backward browsing in my case) but I see it could be generic
so
> > wonder
> >     > if
> >     > > > it is worse thinking about commons or incubator of if scope is
> > too
> >     > small
> >     > > > for that and keeping it specific is saner.
> >     > > >
> >     > > > Anyone has some pointers?
> >     > >
> >     > >
> >     > > --
> >     > >
> >     > >
> >     > > ---
> >     > > Marco Neumann
> >     > > KONA
> >     > >
> >     > > --
> >     > >
> >     > >
> >     > > ---
> >     > > Marco Neumann
> >     > > KONA
> >     > >
> >     >
> >
> >
> >     --
> >
> >
> >     ---
> >     Marco Neumann
> >     KONA
> >
> >
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> > For additional commands, e-mail: dev-help@commons.apache.org
> >
> >



-- 
Matt Sicker <boards@gmail.com>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org


Mime
View raw message