ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maxim Muzafarov <maxmu...@gmail.com>
Subject Re: [DISCUSSION] Channel communication between nodes
Date Fri, 17 May 2019 09:04:26 GMT
Igniters,


I've implemented the file transfer machinery between grid nodes over
Communication SPI covered by JIRA [1] and as the first part of IEP-28
[3]. Please, consider my PR [2] to be reviewed and included in the
Apache Ignite project source code.

These changes will allow developer (and the user in future) uploading
arbitrary files from one cluster node to another. It is a mandatory
feature for the new rebalance by partition files procedure (you can
find all the details of it on IEP-28 [3]) but also can be used, for
example, as a new way of resource deployment (e.g. on
UriDeploymentSPI).

The starting point of PR reviewing and examples of using the file
transfer can be --
IgniteFileTransmitProcessorSelfTest#testFileHandlerBase. Any feedback
is really appreciated. Please, don't be silent.


PR [2] contains two major changes (components):

1) Adding channel support for the Communication SPI, which allows from now:
- establishing channel connections to the remote node to an arbitrary
topic (GridTopic) with predefined channel processing policy;
- listening to incoming channel creation events and registering
connection handlers on the particular node;
- an arbitrary set of channel attributes configured on connection
handshake (exchanging the channel attributes );
The main API changes in the communication component can be found here [5].

2) Creating a new FileTransmitProcessor, which provide an API for
sending and receiving files. It supports:
- using different approaches of incoming data handling – buffered and
direct (zero-copy approach of FileChannel#transferTo);
- transferring data by chunks of predefined size with saving
intermediate results;
- re-establishing connection if an error occurs and continue file
upload\download;
- limiting connection bandwidth (upload and download) at runtime;
The write\read file handlers API can be found here [6].

As we discussed before, I've kept all these as the part of the
internal API, so I'm expecting the merge will be quite safe.

Summary,

JIRA: [1]
PR: [2]
Upsource: [4]

[1] https://issues.apache.org/jira/browse/IGNITE-10619
[2] https://github.com/apache/ignite/pull/5619
[3] https://cwiki.apache.org/confluence/display/IGNITE/IEP-28%3A+Cluster+peer-2-peer+balancing#IEP-28:Clusterpeer-2-peerbalancing-Filetransferbetweennodes
[4] https://reviews.ignite.apache.org/ignite/review/IGNT-CR-1035
[5] https://cwiki.apache.org/confluence/display/IGNITE/IEP-28%3A+Cluster+peer-2-peer+balancing#IEP-28:Clusterpeer-2-peerbalancing-API
[6] https://cwiki.apache.org/confluence/display/IGNITE/IEP-28%3A+Cluster+peer-2-peer+balancing#IEP-28:Clusterpeer-2-peerbalancing-API.1


On Tue, 2 Apr 2019 at 15:37, Nikolay Izhikov <nizhikov@apache.org> wrote:
>
> Hello, Maxim.
>
> Thanks for the update.
>
> Please, let me know when PR will be ready.
> I will do the review.
>
> В Вт, 02/04/2019 в 14:52 +0300, Maxim Muzafarov пишет:
> > Ivan,
> >
> > I tend to agree with you. Let's keep changes as an internal Apache Ignite API.
> >
> > Igniters,
> >
> > I need your support by reviewing the issue [1] with my changes of
> > CommunicationSpi about establishing direct socket connections between
> > cluster nodes. I've prepared the draft PR [2] with such changes. I'll
> > add more JUnit tests to this PR and I'll provide all the
> > implementation details.
> > Can anyone help me with the review?
> >
> > The issue [1] is related to my work over IEP-28 of rebalancing cache
> > data by sending cache partition files (the persistence must be
> > enabled). You can find the rebalancing process details on the IEP-28
> > page [3]. In general, the whole rebalance by partition files process
> > can be divided on five components:
> >  - establishing socket connections ~ 3000 lines (CommunicationSpi)
> >  - create consistent cache partition file ~ 6000 lines (Checkpointer)
> >  - downloading, uploading partition files ~ 4000 lines
> > (CachePartitionUploader, CachePartitionDownloader)
> >  - handling cache puts during the partition file download ~ 3000 lines
> > (CachupWALManager)
> >  - rebuilding indexes ~ 2000 lines
> >
> > All these changes together (single PR) are big and complex, so better
> > to review them step by step. And I'd like to find the support of the
> > community and ask the review of changes in CommunicationSpi module of
> > the Apache Ignite first.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-10619
> > [2] https://github.com/apache/ignite/pull/5619
> > [3] https://cwiki.apache.org/confluence/display/IGNITE/IEP-28%3A+Cluster+peer-2-peer+balancing
> >
> > On Tue, 5 Mar 2019 at 17:52, Павлухин Иван <vololo100@gmail.com>
wrote:
> > >
> > > Maxim,
> > >
> > > My humble opinion. If there is no convenient means to implement
> > > partition file sending today then we should introduce something. And
> > > keeping such facility private is much easier, because introduction of
> > > new public API is a significantly more complex task.
> > >
> > > пт, 1 мар. 2019 г. в 19:44, Maxim Muzafarov <maxmuzaf@gmail.com>:
> > > >
> > > > Igniters,
> > > >
> > > > Apache Ignite has a very suitable messaging user interface [1] for
> > > > topic-based communication between nodes (or a specific group of nodes
> > > > within a cluster). The messaging functionality in Ignite is provided
> > > > via IgniteMessaging interface. It allows:
> > > > - send a message to a certain topic
> > > > - register local\remote listeners
> > > >
> > > > I really like this feature, but the disadvantage here is when the user
> > > > wants to transfer a large amount of binary data (e.g. files) between
> > > > nodes he must create a complex logic to wrap it into messages. I think
> > > > Ignite could have an interface e.g. IgniteChannels which will allow:
> > > > - register local\remote listeners for channel created\destroy events.
> > > > - create a channel connection (a wrapped socket channel) to a certain
> > > > node\group of nodes and the desired topic
> > > >
> > > > As another suitable case where such a feature can be applied is
> > > > internal usage for Apache Ignite needs. I can mention here the task of
> > > > cluster rebalancing by sending cache partition files between nodes.
> > > > I've posted a small description of it on the IEP-28 page [2].
> > > >
> > > >
> > > > WDYT about it?
> > > >
> > > > ---
> > > >
> > > > API (assumed)
> > > >
> > > > IgniteChannels chnls = ignite0.channels();
> > > > chnls.remoteListen(TOPIC.MY_TOPIC, new RemoteListener());
> > > >
> > > > IgniteSocketChannel ch0 = chnls.channel(node, TOPIC.MY_TOPIC);
> > > > ch0.writeInt(bigFile.size());
> > > > ch0.transferTo(FileChannel.open(bigFile.path(), StandardOpenOption.READ))
> > > >
> > > >
> > > > /** */
> > > >
> > > > private class RemoteListener
> > > >     implements IgniteBiPredicate<UUID, IgniteSocketChannel> {
> > > >
> > > >     @IgniteInstanceResource
> > > >     private Ignite ignite;
> > > >
> > > >     @Override public boolean apply(
> > > >         UUID nodeId,
> > > >         IgniteSocketChannel ch
> > > >     ) {
> > > >         int size = ch.readInt();
> > > >         ignite.fileSystem("base")
> > > >             .create("bigfile.mpg")
> > > >             .transferFrom(ch, size);
> > > >         return true;
> > > >     }
> > > > }
> > > >
> > > >
> > > > [1] https://apacheignite.readme.io/docs/messaging
> > > > [2] https://cwiki.apache.org/confluence/display/IGNITE/IEP-28%3A+Cluster+peer-2-peer+balancing#IEP-28:Clusterpeer-2-peerbalancing-CommunicationSpi
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Ivan Pavlukhin

Mime
View raw message