ace-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject Re: Re: Versioned OBR proposal
Date Thu, 09 Feb 2012 14:57:43 GMT
Hello Marcel,

first thanks for the explanation of the workflow, which is really helpful. It's been quite
a while, but I would like to finish the story now so here is another proposal for the integration
of the semantic versioning to ace. 

The main principle of this extension is to perform versioning on all the bundles that are
uploaded, because there is no reason to keep a set of bundles with incorrect versions and
another with correct. I could imagine, this can happen at the beginning of doPost method of
the servlet. The problem is then shrinked to the file names only. As to the REST API, bundles
should be uploaded using the new "POST /?store_bundle" method, which stores the bundle in
the root of the OBR under the "symbolicname-version.jar" file name. User uploads the first
bundle, server stores it only, user uploads the updated bundle (which has typically the same
file name on the client), server performs versioning against the first and stores it under
the file name with new version (which produces no conflict in the file names). However, for
backward compatibility the client will still have a possibility to upload the bundle using
the present "POST /FILENAME" as any other resource and store the bundle under the path, that
he decides. Then, when he ask for this bundle only according to its symbolic name (using "GET
/bsn=BUNDLE_SYMBOLIC_NAME"), we are not sure where the bundle really is on the OBR, as we
can't rely on the file names. One way is to parse the repository.xml all the time and search
for the latest version of the bundle. To get rid of it, BundleStore class will keep a map
symbolicname-bundleinfo(=filename,version) of all the latest bundles and in put(String, data),
we update it if the version of the uploaded bundle after versioning is higher.

Here is the whole proposed REST API:

 => get(String fileName)
        returns the resource from the specified path
 <= HTTP 200 (HttpServletResponse.SC_OK) 
        returns the resource
    HTTP 400 (HttpServletResponse.SC_BAD_REQUEST) 
        if no resource is specified
    HTTP 404 (HttpServletResponse.SC_NOT_FOUND) 
        if the specified resource did not exist
 => getBundle(String symbolicName)
        gets the latest bundle
        1. lookup the fileName in the map symbolicname-bundleinfo
	2. get(String fileName)
 <= HTTP 200 (HttpServletResponse.SC_OK) 
        returns the bundle
	X-BundleVersion header set
    HTTP 404 (HttpServletResponse.SC_NOT_FOUND) 
        if no bundles with the specified symbolic name found
GET /?bsn=BUNDLE_SYMBOLIC_NAME&amp;v=[1.0,2.0)
 => getBundle(String symbolicName, VersionRange range)
        gets the latest bundle in the range
	1. update repository.xml
	2. parse repository.xml for the latest version in the range
	3. get(String fileName)
 <= HTTP 200 (HttpServletResponse.SC_OK) 
        returns the bundle
        X-BundleVersion header set
    HTTP 404 (HttpServletResponse.SC_NOT_FOUND) 
        if the specified bundle did not exits
 (Note: Client can also do get("repository.xml") + parsing 
       + searching for a bundle according to its symbolic name 
       on its own)
 => 1. if (bundle) 
           data=performVersioning(InputStream data)
               1.1 lookup the symbolic name from the stream
               1.2 latestInOBR = getBundle(String symbolicName)
               1.3 service.createVersionedBundle(
	               InputStream data, InputStream latestInOBR)
	       1.4 check the version and if newest, 
		   insert into map symbolicname-bundleinfo
    2. put(String fileName, InputStream data)
           stores the resource under specified filename
 <= HTTP 200 (HttpServletResponse.SC_OK) 
        storing OK
    HTTP 409 (HttpServletResponse.SC_CONFLICT) 
        if the resource already exists
    HTTP 500 (HttpServletResponse.SC_INTERNAL_SERVER_ERROR) 
        if there was a problem storing the resource
 (Note: If a bundle is put to OBR by another way than REST, 
       versioning isn't performed)

POST /?store_bundle
 => if (bundle) 
        1. data=performVersioning(data)
               see higher
        2. fileName=generateFileName(data)
        3. put(fileName, data)
        4. redirect
    else {
 <= HTTP 303 (HttpServletResponse.SC_SEE_OTHER) 
        storing OK 
    HTTP 400 (HttpServletResponse.SC_BAD_REQUEST) 
        if resource isn't a bundle
 => remove(String fileName)
        deletes the resource at the specified path
 <= HTTP 200 (HttpServletResponse.SC_OK)
    HTTP 400 (HttpServletResponse.SC_BAD_REQUEST) 
	if no resource is specified
    HTTP 404 (HttpServletResponse.SC_NOT_FOUND) 
        if the specified resource did not exist
    HTTP 500 (HttpServletResponse.SC_INTERNAL_SERVER_ERROR) 
        if there was a problem deleting the resource
 => 1. update the repository.xml
    2. parse repository.xml for all the versions
    3. remove(String fileName) 
           delete in loop all the bundles with this symbolic name
 <= HTTP 200 (HttpServletResponse.SC_OK) 
    HTTP 404 (HttpServletResponse.SC_NOT_FOUND) 
        if no bundles with the specified symbolic name found
    HTTP 500 (HttpServletResponse.SC_INTERNAL_SERVER_ERROR) 
        if there was a problem deleting the bundles
DELETE /?bsn=BUNDLE_SYMBOLIC_NAME&amp;v=[1.0,2.0]
 => 1. update the repository.xml
    2. parse repository.xml for all the versions in the range
    3. remove(String fileName) 
           delete in loop all the bundles with this symbolic name
 <= HTTP 200 (HttpServletResponse.SC_OK) 
    HTTP 404 (HttpServletResponse.SC_NOT_FOUND) 
        if no bundles found
    HTTP 500 (HttpServletResponse.SC_INTERNAL_SERVER_ERROR) 
        if there was a problem deleting the bundles
 (Note: user-centric asymetricity
       "GET /?bsn=" returns the latest x "DELETE /?bsn=" deletes all)
I'm looking forward for your ideas.
Best Regards
Jan Zeman

> ------------ Původní zpráva ------------
> Od: Marcel Offermans <>
> Předmět: Re: Versioned OBR proposal
> Datum: 04.12.2011 23:38:26
> ----------------------------------------
> Hello Jan,
> On Dec 1, 2011, at 0:05 AM, <> <> wrote:
> > as a member of a research group in Pilsen, that strives to create 
> > a tool for semantic version of OSGi bundles (,  
> > I have some questions to ask:
> > 
> > I don't think OBR from projects ace-obr-servlet and ace-obr-storage 
> > is accessed via the Vaadin UI interface. Instead of it, as deep 
> > as I've dug, it looks like it uses the ArtifactRepository from 
> > ace-client-repository-* projects. Does this mean, the projects ace-obr 
> > projects are obsolete? Or is the ace-obr-servlet really mapped to 
> > http://localhost:8080/obr only without the web interface? How can one 
> > then manage this OBR, e.g. put the bundles there?
> To start with your first question: no, the OBR bundles are not obsolete at all.
> To explain how and where they're used, I should perhaps first explain a bit more
> about the ACE server and which major subsystems it includes. In this context,
> there are three subsystems which are relevant:
> 1) the OBR bundles
> 2) the client bundles
> 3) the (other) server bundles
> These three subsystems can actually be deployed on different nodes, even though
> by default we all ship them as part of the ACE server.
> The OBR bundles, when deployed with an HTTP server, form a stand-alone OBR which
> publishes (as per OBR spec) the contents of its repository via the
> (automatically generated and updated) repository.xml and extends the normal spec
> a bit because we also implemented an HTTP POST call to add new artifacts to the
> OBR. For convenience, if you have physical access to the filesystem that OBR
> runs on, you can even directly copy artifacts into the "store" folder and they
> will be detected as well (even though we don't actively encourage that, the
> official way is via the POST method as that allows us to properly check the
> contents of the artifact).
> The client bundles, together with the Vaadin web UI and an HTTP server, can also
> be deployed on a different node. Actually, the client bundles form an API that
> can be used in all kinds of clients. Before donating the codebase to Apache, for
> example, we had a Swing based "rich client" that could run on a different
> machine and that communicated with both the ACE server and the OBR. This is also
> why you see that artifacts "enter the system" via the client API. If you dig
> deeper, you will see that from the client, an artifact will be "uploaded" to the
> OBR as well, via a POST call.
> Finally, the other server bundles form the heart of the ACE server and contain
> the major repositories that hold all historic metadata for the store repository
> (which keeps track of the relations between artifacts, features and
> distributions), the license repository (which keeps track of the relations
> between distributions and targets) and the deployment repository (which is
> really the sum of the first two, and maps targets to artifacts in a format that
> is suited for fast deployment). During deployment, when a management agent
> contacts the server and asks for a deployment package, this package is actually
> created on the fly by the server. During that process, it will fetch artifacts
> from the OBR (so that's another place where the OBR is used).
> > Some time ago, our leader Premek Brada proposed adding our semantic 
> > versioning to the OBR of ACE. We would like to go on with this goal. Some 
> > changes were made and I'd like to present it to the you and ask for your 
> > opinion :
> > 
> > 1. PUT
> > Our vision is that by putting a new resource to the OBR, there 
> > would be a decision, whether it is or it isn't a bundle. If it is another
> file, 
> > store it only in the OBR. If it is a bundle, perform the versioning according
> > the last version of the bundle in the OBR first, then store it. This should
> > happen all the time and thus, we could have the consistent list of all the 
> > versions of the bundle, that were uploaded in time. Naturally, if there is no
> > another version, no versioning is performed. 
> If you look at the definition of a REST API then the semantics for PUT are to
> always store (and replace) the exact resource that you specified, so I would not
> want to deviate from that unless we have a really good reason. We are however
> free to add a POST command that uses semantic versioning.
> > The problem is, that only the method put(String fileName, InputStream 
> > data) is present so far, the client decides the filename. If there is a file
> with 
> > this name in the OBR (which is the case by different bundle versions) it would
> > be overwritten.
> We are free to extend our API.
> > To let the OBR API backward compatible and still take advance 
> > from the versioning, the client have to manually add the the version suffix to
> the 
> > filename, but it doesn't know, whether the version in the manifest is really 
> > correct, since no semantic versioning was performed yet. 
> Right now we assume the client has correctly done semantic versioning. Most of
> the time, he will have to do that by hand, but there are other tools surfacing
> that help (BndTools now allows this).
> > We propose to add another method to the OBR servlet like 
> > dryRun(InputStream data), that perform the versioning on the uploaded bundle 
> > but doesn't store it to the OBR, only returns in the HTTP header the info,
> whether 
> > there is another version and if so, also the corrected version of the uploaded
> > bundle. Then the client constructs the filename and calls the common put 
> > method.
> Let me make a counter-proposal here to add a POST method that will do the
> upload, then modify the bundle (if there are existing versions in the OBR of
> this same bundle) and finally either redirect or return the proper URL of the
> versioned bundle.
> > 2. GET
> > Typically I like to get the latest version of the bundle, that is in the
> repository.
> > We can use the present get(filename) method, but we have to know the
> filename.
> > listBundles(String symbolicName) would do the work, possibly by inspecting
> > the repository.xml.
> The current OBR spec indeed assumes that you first GET the repository.xml, parse
> that and based on the information inside that file, make up your mind about what
> exact bundle (or artifact) you want.
> On top of that, I could live with an optional query mechanism that allows you to
> do things like:
> GET /obr/? - to return the highest version of a bundle
> with the specified symbolic name
> GET /obr/?;v=[1.0,2.0) - same, but highest version within
> the given range
> GET /obr/?;v=[1.0,2.0)&amp;lowest - same, but lowest
> within the given range
> or even, some kind of query that will return a list or subset (in which case it
> might actually make sense to either return just URLs or a filtered
> repository.xml)
> Greetings, Marcel

Jan Zeman

View raw message