ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Romain Gilles <romain.gil...@gmail.com>
Subject Re: OSGi migration may require a special marshaller
Date Wed, 04 Nov 2015 17:07:37 GMT
Hi Dmitriy,
I found this post that explain how to find a bundle based on its bundle
name and version.
This post explain for the past to now how to do it in the standard way with
"pull" approach: https://www.eclipse.org/forums/index.php/t/205719/
Regarding the version of OSGi you want to support then some solutions will
works some others will not.
There is an other way to do this stuff without using those "pull" style
approach based on BundleTracker. If you want I can give you the code to do
it with BundlTracker. I think with this solution you will support a wider
range of OSGi version.

Romain


Le mer. 4 nov. 2015 à 04:07, Dmitriy Setrakyan <dsetrakyan@apache.org> a
écrit :

> Andrey, et al,
>
> Can you provide a way on how to get a required Bundle object based on, say,
> bundle symbolic name and version?
>
> I have reached the end of the internet trying to find a way in OSGI to look
> up a Bundle based on something other than an actual Class belonging to that
> bundle, but no luck.
>
> D.
>
> On Tue, Nov 3, 2015 at 5:15 PM, Andrey Kornev <andrewkornev@hotmail.com>
> wrote:
>
> > Dmitriy,
> >
> > I think your approach will work, but I let Romain respond.
> >
> > Also, in terms of the implementation, please keep in mind that the
> > resolver must be called for each non-JDK and non-Ignite core class (it
> > would probably make sense to eschew storing class loaders for such
> classes
> > in favor of compactness of the serialized representation -- see below).
> > Also, it's worth keeping a cache of already resolved class loaders per
> > marshaller invocation (this is probably the context that Romain has
> > mentioned in his previous posting) to minimize the number of the resolver
> > calls.
> >
> > In terms of the resolver's implementation, the simplest way to serialize
> > the class loader would be by capturing two pieces of information (both
> > strings): the bundle symbolic name and the bundle version. This approach
> > however may result in bloating of the serialized representation: I'd
> > roughly estimate the overhead per element to be at least 20-30 bytes (the
> > length of the symbolic name string, plus the length of the version
> string).
> > There are way to reduce the overhead (like serializing the hash code of
> the
> > bundle id string rather than the string itself, and then come up with a
> > clever way of resolving the hash collisions), but they all come at cost
> of
> > increased complexity...
> >
> > An alternative approach would be rely on the special bundle id which is
> an
> > integer and is generated by the OSGi container. But in this case, all
> nodes
> > must ensure that all the bundles have consistent ids (bundle A with id 42
> > on node N1, has the same id on every other node) which is not easy --
> while
> > not entirely impossible -- to guarantee. As long as the nodes are
> > homogeneous (have the same set of bundles deployed) the OSGi container is
> > guaranteed to assign to the bundles the same ids.
> >
> > Thanks
> > Andrey
> >
> > > From: dsetrakyan@apache.org
> > > Date: Tue, 3 Nov 2015 16:29:41 -0800
> > > Subject: Re: OSGi migration may require a special marshaller
> > > To: dev@ignite.apache.org
> > >
> > > Romain,
> > >
> > > In the upcoming release we will be deprecating the OptimizedMarshaller
> > and
> > > will be switching to a default internal marshaller (which is based on
> the
> > > new PortableMarshaller donated by GridGain).
> > >
> > > Having said that, we may be able to pass BinaryWriter and BinaryReader
> > > instead of byte arrays. This will be pretty close to passing the
> stream,
> > as
> > > suggested by Andrey.
> > >
> > > Also, I still think that we should only resolve class loaders and not
> the
> > > class itself. The main reason is that Ignite will encode class names
> into
> > > an integer hash code and will store the integer->class-fqn mapping in
> > > internal replicated cache. I doubt users will get more efficient than
> an
> > > integer (4 bytes) for a class name.
> > >
> > > On the receiving side, once we are able to get the right class loader,
> we
> > > can easily get the proper class by calling
> > ClassLoader.findClass(class-fqn).
> > >
> > > Thoughts?
> > >
> > > D.
> > >
> > > On Tue, Nov 3, 2015 at 2:01 PM, Romain Gilles <romain.gilles@gmail.com
> >
> > > wrote:
> > >
> > > > Hi,
> > > > Maybe a missing point. I think but I'm not sure that in the
> > > > OptimizedMarshaller there is a caching of already serialized classes.
> > Due
> > > > to the dynamic nature of OSGi this may lead to memory leak. In fact
> if
> > a
> > > > bundle is refreshed, it will produce a new BundleRevision and
> > therefore a
> > > > new classloader. And if you don't release the class from the previous
> > > > BundleRevision then you endup with memory leak. So maybe the
> Marshaller
> > > > interface or somewhere should provide a way to free those classes /
> > > > classloaders.
> > > >
> > > > Regards,
> > > >
> > > > Romain
> > > >
> > > > Le mar. 3 nov. 2015 à 22:42, Andrey Kornev <andrewkornev@hotmail.com
> >
> > a
> > > > écrit :
> > > >
> > > > > Romain/Dmitriy,
> > > > >
> > > > > I prefer Romain's approach, but just curious, in the API you guys
> are
> > > > > proposing why use a byte[] rather than OutputStream/InputStream?
> > With a
> > > > > byte[], one would inevitably end up wrapping it into a byte stream
> > class.
> > > > > Also, the stream-based interface would be more in line with the
> > > > Marshaller
> > > > > API.
> > > > >
> > > > > Also for symmetry with the readClass() method, I suggest the
> > writeClass()
> > > > > take a Class<?> rather than an object.
> > > > >
> > > > > Regards
> > > > > Andrey
> > > > >
> > > > > > From: romain.gilles@gmail.com
> > > > > > Date: Tue, 3 Nov 2015 21:24:01 +0000
> > > > > > Subject: Re: OSGi migration may require a special marshaller
> > > > > > To: dev@ignite.apache.org
> > > > > >
> > > > > > Hi Dmitriy,
> > > > > > I think your solution is good. Maybe I will change it a little
> > bit...
> > > > :P
> > > > > > I think you should delegate the Class resolution to the resolver.
> > > > Because
> > > > > > for example with your current solution the marshaller may before
> > (or
> > > > > after)
> > > > > > store the fqn of the class (maybe only the first time it
> encounter
> > it)
> > > > > but
> > > > > > in order to save the classloader context resolution the
> > implementation
> > > > of
> > > > > > the resolver may have to save the package name of the given
> object
> > and
> > > > > some
> > > > > > extra info therefore the content package name will be serialized
> > two
> > > > > times.
> > > > > > Ok, it's not a big deal.
> > > > > > But now if the resolver identify that it can reuse the same
class
> > > > loader
> > > > > > for a couple of classes. It will be interesting for it to have
a
> > > > > > serialization context in order to save, let say, classloader/id
> > mapping
> > > > > in
> > > > > > order to save the id instead of a more longer representation.
> > > > > > So I propose something like that:
> > > > > > *public interface ClassResolver {*
> > > > > > *    // This method is invoked on the sender side to *
> > > > > > *    // marshal the information about the class.*
> > > > > > *    // where the context is a map style object that is reset/new
> > for
> > > > > each
> > > > > > object graph serialization.*
> > > > > > *    public byte[] writeClass(Object o, Context context) throws
> > > > > > IgniteException;*
> > > > > >
> > > > > > *    // This method is invoked on the receiving side to*
> > > > > > *    // retrieve the class based on provided information.*
> > > > > > *    // where the context is a map style object that is reset/new
> > for
> > > > > each
> > > > > > object graph serialization.*
> > > > > > *    public Class<?> readClass(byte[], Context context)
throws
> > > > > > IgniteException;*
> > > > > > *}*
> > > > > >
> > > > > > I think your proposal will solve our issue and maybe also open
a
> > door
> > > > for
> > > > > > the osgi development.
> > > > > > Let me know what do you think about me proposal? :)
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Romain
> > > > > >
> > > > > > Le mar. 3 nov. 2015 à 20:05, Dmitriy Setrakyan <
> > dsetrakyan@apache.org>
> > > > a
> > > > > > écrit :
> > > > > >
> > > > > > > Romain,
> > > > > > >
> > > > > > > Can you comment on the ClassLoaderResolver suggestion that
I
> > proposed
> > > > > > > earlier? Will it solve your problem?
> > > > > > >
> > > > > > > D.
> > > > > > >
> > > > > > > On Tue, Nov 3, 2015 at 2:38 AM, Romain Gilles <
> > > > romain.gilles@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Raul,
> > > > > > > >  - Do you plan to use the TCCL when marshalling in
OSGi env?
> > If yes
> > > > > how?
> > > > > > > >  - I like the point 2. Maybe for a graph of object
that come
> > from
> > > > > > > different
> > > > > > > > packages / bundles you may have to recursively capture
the
> > package
> > > > > > > version
> > > > > > > > of the individual element of your graph.
> > > > > > > >  - For the point 3 I wonder if it will not over-complexify
> the
> > > > > solution.
> > > > > > > As
> > > > > > > > a developer you will have to think about it. And it
is not
> > obvious
> > > > > in the
> > > > > > > > code where things are serialized or not. You may use
lambda
> in
> > your
> > > > > > > > application code therefore the current package become
what
> you
> > call
> > > > > a DTO
> > > > > > > > package. The current state of ignite does not enforce
you to
> > > > specify
> > > > > it
> > > > > > > for
> > > > > > > > "classical" classloading application. If you introduce
this
> > extra
> > > > > step
> > > > > > > for
> > > > > > > > OSGi ready application you will maybe discourage people
to
> use
> > > > OSGi.
> > > > > > > >
> > > > > > > > To comeback to our solution. We start we a strong
assumption:
> > our
> > > > > cluster
> > > > > > > > is homogeneous in term of code so, of course, it simplify
our
> > life
> > > > > :).
> > > > > > > >
> > > > > > > > BTW if you can introduce an extension point in the
class
> > resolution
> > > > > > > > mechanism it can be interesting for end user in order
to
> allow
> > them
> > > > > to
> > > > > > > > optimize it based on there specific use cases.
> > > > > > > >
> > > > > > > > Romain.
> > > > > > > >
> > > > > > > > Le mar. 3 nov. 2015 à 00:21, Raul Kripalani <
> raul@evosent.com>
> > a
> > > > > écrit :
> > > > > > > >
> > > > > > > > > Hi Andrey,
> > > > > > > > >
> > > > > > > > > Thanks for the participation in this topic.
> > > > > > > > >
> > > > > > > > > I don't like the solution to incorporate the
bundle
> symbolic
> > name
> > > > > in
> > > > > > > the
> > > > > > > > > serialised form. Nothing guarantees that the
classdef will
> be
> > > > > located
> > > > > > > > under
> > > > > > > > > the same bundle in both source and target machines.
We also
> > have
> > > > to
> > > > > > > take
> > > > > > > > > into account that OSGi allows for multiple versions
of the
> > same
> > > > > > > > > bundle/packages to co-exist in the same  container.
So it
> > becomes
> > > > > more
> > > > > > > > > complex.
> > > > > > > > >
> > > > > > > > > Using the TCCL should work when serialising,
but it'll
> > probably
> > > > be
> > > > > of
> > > > > > > no
> > > > > > > > > use when deserialising on the other end.
> > > > > > > > >
> > > > > > > > > I need to enhance Ignite to:
> > > > > > > > >
> > > > > > > > > 1. Use the TCCL when marshalling on one end.
> > > > > > > > > 2. Incorporate the package version of the class
in the
> > serialised
> > > > > form
> > > > > > > if
> > > > > > > > > Ignite is running in an OSGi environment.
> > > > > > > > > 3. On the receiver end, discover cache entities
/ DTOs in
> all
> > > > > bundles
> > > > > > > > > through a custom OSGi manifest header or the
like, as I
> > explained
> > > > > > > before.
> > > > > > > > > Package version must be taken into account.
> > > > > > > > >
> > > > > > > > > What do you think?
> > > > > > > > >
> > > > > > > > > Raúl.
> > > > > > > > > On 2 Nov 2015 17:25, "Andrey Kornev" <
> > andrewkornev@hotmail.com>
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Raul,
> > > > > > > > > >
> > > > > > > > > > The fundamental hurdle we need to jump over
to make
> Ignite
> > > > > > > OSGi-enabled
> > > > > > > > > is
> > > > > > > > > > the marshalling. More specifically the issue
is with
> > > > > deserialization
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > classes that are provided by the bundles
other than the
> > Ignite
> > > > > bundle
> > > > > > > > > > itself.
> > > > > > > > > >
> > > > > > > > > > When the Ignite transport layer receives
a message it
> > needs to
> > > > > figure
> > > > > > > > out
> > > > > > > > > > how to deserialize the bytes and for that
it needs to
> know
> > the
> > > > > bundle
> > > > > > > > > that
> > > > > > > > > > provides the class to be deserailized. At
this point TCCL
> > is of
> > > > > no
> > > > > > > use.
> > > > > > > > > To
> > > > > > > > > > make things more complex, the class may
contain other
> > classes
> > > > > that
> > > > > > > come
> > > > > > > > > > from other bundles, and so on recursively.
This means
> that
> > each
> > > > > > > object
> > > > > > > > in
> > > > > > > > > > the hierarchy must be serialized with its
bundle name (or
> > > > bundle
> > > > > id),
> > > > > > > > so
> > > > > > > > > > that the deserializer will then be able
to correctly
> > resolve
> > > > the
> > > > > > > class
> > > > > > > > > > while traversing the object hierarchy during
> > deserialization.
> > > > > > > > > >
> > > > > > > > > > Unfortunately, Ignite's OptimizedMarshaller
is lacking
> the
> > > > > ability to
> > > > > > > > > plug
> > > > > > > > > > in a custom class resolver. Romain's solution
was to use
> > Kryo
> > > > > that
> > > > > > > does
> > > > > > > > > > provide a way to customize class resolution.
It has
> solved
> > the
> > > > > > > problem
> > > > > > > > of
> > > > > > > > > > capturing the bundle info and he was able
to successfully
> > run
> > > > > Ignite
> > > > > > > > as a
> > > > > > > > > > bundle in an OSGi container (after some
repackaging and
> > > > > inclusion of
> > > > > > > > the
> > > > > > > > > > manifest). But Kryo-based marshalling introduced
a lot of
> > > > > complexity
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > code and incorrect use of Kryo's numerous
serializers
> > caused
> > > > some
> > > > > > > weird
> > > > > > > > > > hard-to-debug issues in the Ignite core
(like duplicate
> > cache
> > > > > entries
> > > > > > > > due
> > > > > > > > > > to incorrect marshalling of the GridDhtPArtitonFullMap
> > class --
> > > > > go
> > > > > > > > > > figure!). Overall the Kryo-based solution
is brittle and
> > hard
> > > > to
> > > > > > > > > maintain.
> > > > > > > > > >
> > > > > > > > > > I feel the correct solution to OSGi problem
would be to
> > > > > > > > > > 1) enhance the OptimizedMarshaller to allow
custom class
> > > > > resolution.
> > > > > > > > > > 2) provide an OSGi-enabled OptimizedMarshaller
(in
> > addition to
> > > > > the
> > > > > > > > > > original one) to be used in OSGi environment.
> > > > > > > > > >
> > > > > > > > > > Regards
> > > > > > > > > > Andrey
> > > > > > > > > >
> > > > > > > > > > > From: raulk@apache.org
> > > > > > > > > > > Date: Mon, 2 Nov 2015 12:41:47 +0000
> > > > > > > > > > > Subject: Re: OSGi migration may require
a special
> > marshaller
> > > > > > > > > > > To: dev@ignite.apache.org
> > > > > > > > > > >
> > > > > > > > > > > Hi Romain,
> > > > > > > > > > >
> > > > > > > > > > > I'm working on the OSGi compatibility
of Ignite. I
> > appreciate
> > > > > your
> > > > > > > > > input.
> > > > > > > > > > >
> > > > > > > > > > > I'm thinking about the situation you
describe and I
> > wonder if
> > > > > > > you're
> > > > > > > > > > > exporting Ignite as an OSGi service
which is then
> > consumed
> > > > from
> > > > > > > other
> > > > > > > > > > > bundles. Under this situation, it would
be quite easy
> to
> > > > > reproduce
> > > > > > > > the
> > > > > > > > > > > behaviour you describe if Ignite is
not resolving
> > classes via
> > > > > the
> > > > > > > > TCCL.
> > > > > > > > > > > Need to dig deeper into that.
> > > > > > > > > > >
> > > > > > > > > > > Off the top of my head, there are two
alternatives to
> > solve
> > > > it:
> > > > > > > > > > >
> > > > > > > > > > > 1. Use the TCCL for marshalling/unmarshalling
(if not
> > already
> > > > > > > used) –
> > > > > > > > > we
> > > > > > > > > > > gotta be wary of possible regressions.
> > > > > > > > > > > 2. Create a special OSGi header Ignite-Export-Package
> so
> > that
> > > > > > > bundles
> > > > > > > > > > > containing DTOs can expose packages
to Ignite's
> > marshallers.
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > >
> > > > > > > > > > > *Raúl Kripalani*
> > > > > > > > > > > PMC & Committer @ Apache Ignite,
Apache Camel |
> > Integration,
> > > > > Big
> > > > > > > Data
> > > > > > > > > and
> > > > > > > > > > > Messaging Engineer
> > > > > > > > > > > http://about.me/raulkripalani |
> > > > > > > > > http://www.linkedin.com/in/raulkripalani
> > > > > > > > > > > http://blog.raulkr.net | twitter: @raulvk
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Nov 2, 2015 at 9:56 AM, Gilles,
Romain <
> > > > > > > > > romain.gilles@misys.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > I'm really interested in this
issue:
> > > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-1270
.
> We
> > > > some
> > > > > > > stuff
> > > > > > > > to
> > > > > > > > > > make
> > > > > > > > > > > > it work in our osgi environment.
The main issue for
> us
> > now
> > > > > is the
> > > > > > > > > > > > serialization. I think it you
will have to rework the
> > > > > > > > > > OptimizedMarshaller
> > > > > > > > > > > > or any other marshaller that works
with object that
> > come
> > > > from
> > > > > > > > outside
> > > > > > > > > > your
> > > > > > > > > > > > class space.
> > > > > > > > > > > >
> > > > > > > > > > > > We have try kryo that works. Kryo
provide an
> extension
> > > > point
> > > > > in
> > > > > > > > order
> > > > > > > > > > to
> > > > > > > > > > > > resolve the classes:
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> >
> https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/ClassResolver.java
> > > > > > > > > > > > . With this extension we are able
to solve the
> problem
> > of
> > > > > > > external
> > > > > > > > > > classes.
> > > > > > > > > > > > The only issue with kryo is that
some classes need a
> > > > certain
> > > > > care
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > serialization process and therefore
a specialized
> > > > serializer.
> > > > > > > > > > > >
> > > > > > > > > > > > So I would like to know from the
community what do
> > think of
> > > > > > > > changing
> > > > > > > > > > the
> > > > > > > > > > > > way the optimized marshaller works
or introducing the
> > > > > support of
> > > > > > > > yet
> > > > > > > > > > > > another marshaller based on a
kryo like technology?
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks in advance,
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Romain.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > PS: I'm ready to help in the both
cases.
> > > > > > > > > > > > "Misys" is the trade name of the
Misys group of
> > companies.
> > > > > This
> > > > > > > > email
> > > > > > > > > > and
> > > > > > > > > > > > any attachments have been scanned
for known viruses
> > using
> > > > > > > multiple
> > > > > > > > > > > > scanners. This email message is
intended for the
> named
> > > > > recipient
> > > > > > > > > only.
> > > > > > > > > > It
> > > > > > > > > > > > may be privileged and/or confidential.
If you are not
> > the
> > > > > named
> > > > > > > > > > recipient
> > > > > > > > > > > > of this email please notify us
immediately and do not
> > copy
> > > > > it or
> > > > > > > > use
> > > > > > > > > > it for
> > > > > > > > > > > > any purpose, nor disclose its
contents to any other
> > person.
> > > > > This
> > > > > > > > > email
> > > > > > > > > > does
> > > > > > > > > > > > not constitute the commencement
of legal relations
> > between
> > > > > you
> > > > > > > and
> > > > > > > > > > Misys.
> > > > > > > > > > > > Please refer to the executed contract
between you and
> > the
> > > > > > > relevant
> > > > > > > > > > member
> > > > > > > > > > > > of the Misys group for the identity
of the
> contracting
> > > > party
> > > > > with
> > > > > > > > > > which you
> > > > > > > > > > > > are dealing.
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message