ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexey Zinoviev <zaleslaw....@gmail.com>
Subject Re: [ML] Deployment of user-defined preprocessors
Date Sun, 08 Sep 2019 07:47:35 GMT
Thank you for the contribution, will make a review this week

вс, 8 сент. 2019 г., 10:05 Алексей Платонов <aplatonovv@gmail.com>:

> Hi, Igniters!
> I've prepared a PR with fixes to correct peer class loading for ML-lambdas.
> Here it is: https://github.com/apache/ignite/pull/6853
>
> It will be great if someone reviews my PR. Alexey, Yuriy, could you please
> to see this contribution.
>
> Thanks!
>
> Best regards.
> Alexey Platonov
>
> ср, 5 июн. 2019 г. в 17:20, Yuriy Babak <y.chief@gmail.com>:
>
> > Alexey,
> >
> > This is a cool change, do you create a ticket for it?
> >
> > If no I can create one.
> >
> > Best regards,
> > Yuriy Babak
> >
> >
> > пт, 31 мая 2019 г. в 14:20, Алексей Платонов <aplatonovv@gmail.com>:
> >
> > > Hi, Igniters!
> > > Currently we don't have an ability to deploy automatically user-defined
> > > preprocessors and vectorizers. Client's code should be deployed
> manually
> > to
> > > Ignite server nodes.
> > >
> > > I have an idea how to fix it. If we pass user's classloader and one of
> > > user-defined classes from fit-level to
> > > ComputeUtils.affinityCallWithRetries() then we wiil be able to use
> > > GridPeerDeployAware interface to send informtation about this
> classloader
> > > to server nodes.
> > >
> > > To support this ability we can define interfaces like these:
> > >
> > > public interface DeployableObject {
> > >     public List<Object> getDependencies();
> > > }
> > >
> > > and
> > >
> > > public interface DeployingContext {
> > >     public Class<?> userClass();
> > >     public ClassLoader clientClassLoader();
> > > }
> > >
> > > DeployableObject will be mark for our ignite-ml final classes like
> > trainers
> > > or concrete preprocessors and it can be able to return all dependencies
> > > that should be deployed to server nodes if it's needed. If these
> > > dependencies are DeployableObjects too then depenndencies will be
> > unfolded
> > > recursively. Classes that isn't defined as DeployableObject will be
> > > recognized as user-defined (NOTE: all leaf classes in our hierarchy
> will
> > be
> > > DeployableObject).
> > >
> > > This list of DeployableObjects will be user for define user class
> loader
> > > and one of these objects will be used for passing to
> GridPeerDeployAware.
> > >
> > > So, this logic allows to pass user-defined Preprocessors and
> Vectorizers
> > to
> > > training algorithms and pipelines.
> > >
> > > What do you think?
> > >
> > > Sincerely
> > > Alexey Platonov
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message