ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ivan Daschinsky <ivanda...@gmail.com>
Subject Re: IEP-61 Technical discussion
Date Wed, 25 Nov 2020 18:38:24 GMT
Alexey, I kindly ask you to move the meeting a little bit earlier, ideal
variant -- in the morning.

ср, 25 нояб. 2020 г. в 20:10, Alexey Goncharuk <alexey.goncharuk@gmail.com>:

> Folks, let's have the call on Friday, Nov 27th at 18:00 MSK? We can use the
> following waiting room link:
>  https://zoom.us/j/99450012496?pwd=RWZmOGhCNWlRK0ZpamdOOTZsYTJ0dz09
>
> Let me know if this time works for everybody.
>
> ср, 25 нояб. 2020 г. в 16:42, Alexey Goncharuk <alexey.goncharuk@gmail.com
> >:
>
> > Folks,
> >
> > I've made some edits in IEP-61 [1] regarding the group membership service
> > and transaction protocol interaction with the replication infrastructure,
> > please take a look before our Friday call.
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-61%3A+Common+Replication+Infrastructure
> >
> > пн, 23 нояб. 2020 г. в 13:28, Alexey Goncharuk <
> alexey.goncharuk@gmail.com
> > >:
> >
> >> Thanks, Ivan,
> >>
> >> Another protocol for group membership worth checking out is RAPID [1] (a
> >> recent one). Not sure though if there are any available implementations
> for
> >> it already.
> >>
> >> [1]
> https://www.usenix.org/system/files/conference/atc18/atc18-suresh.pdf
> >>
> >> пн, 23 нояб. 2020 г. в 10:46, Ivan Daschinsky <ivandasch@gmail.com>:
> >>
> >>> Also, here is some interesting reading about gossip, SWIM etc.
> >>>
> >>> 1 --
> >>> http://www.cs.cornell.edu/Info/Projects/Spinglass/public_pdfs/SWIM.pdf
> >>> 2 --
> >>>
> >>>
> http://www.antonkharenko.com/2015/09/swim-distributed-group-membership.html
> >>> 3 -- https://github.com/hashicorp/memberlist (Foundation library of
> >>> hashicorp serf)
> >>> 4 -- https://github.com/scalecube/scalecube-cluster -- (Java
> >>> implementation
> >>> of SWIM)
> >>>
> >>> чт, 19 нояб. 2020 г. в 16:35, Ivan Daschinsky <ivandasch@gmail.com>:
> >>>
> >>> > >> Friday, Nov 27th work for you? If ok, let's have an open call
> then.
> >>> > Yes, great
> >>> > >> As for the protocol port - we will not be dealing with the
> >>> > concurrency...
> >>> > >>Judging by the Rust port, it seems fairly straightforward.
> >>> > Yes, they chose split transport and logic. But original Go package
> from
> >>> > etcd (see raft/node.go) contains some  heartbeats mechanism etc.
> >>> > I agree with you, this seems not to be a huge deal to port.
> >>> >
> >>> > чт, 19 нояб. 2020 г. в 16:13, Alexey Goncharuk <
> >>> alexey.goncharuk@gmail.com
> >>> > >:
> >>> >
> >>> >> Ivan,
> >>> >>
> >>> >> Agree, let's have a call to discuss the IEP. I have some more
> thoughts
> >>> >> regarding how the replication infrastructure works with
> >>> >> atomic/transactional caches, will put this info to the IEP. Does
> next
> >>> >> Friday, Nov 27th work for you? If ok, let's have an open call then.
> >>> >>
> >>> >> As for the protocol port - we will not be dealing with the
> concurrency
> >>> >> model if we choose this way, this is what I like about their code
> >>> >> structure. Essentially, the raft module is a single-threaded
> automata
> >>> >> which
> >>> >> has a callback to process a message, process a tick (timeout) and
> >>> produces
> >>> >> messages that should be sent and log entries that should be
> persisted.
> >>> >> Judging by the Rust port, it seems fairly straightforward. Will
be
> >>> happy
> >>> >> to
> >>> >> discuss this and other alternatives on the call as well.
> >>> >>
> >>> >> чт, 19 нояб. 2020 г. в 14:41, Ivan Daschinsky <ivandasch@gmail.com
> >:
> >>> >>
> >>> >> > > Any existing library that can be used to avoid re-implementing
> the
> >>> >> > protocol ourselves? Perhaps, porting the existing implementation
> to
> >>> Java
> >>> >> > Personally, I like this idea. Go libraries (either raft module
of
> >>> etcd
> >>> >> or
> >>> >> > serf by Hashicorp) are famous for clean code, good design,
> >>> stability,
> >>> >> not
> >>> >> > enormous size.
> >>> >> > But, on other side, Go has different model for concurrency
and
> >>> porting
> >>> >> > probably will not be so straightforward.
> >>> >> >
> >>> >> >
> >>> >> >
> >>> >> > чт, 19 нояб. 2020 г. в 13:48, Ivan Daschinsky <
> ivandasch@gmail.com
> >>> >:
> >>> >> >
> >>> >> > > I'd suggest to discuss this IEP and technical details
in open
> ZOOM
> >>> >> > > meeting.
> >>> >> > >
> >>> >> > > чт, 19 нояб. 2020 г. в 13:47, Ivan Daschinsky
<
> >>> ivandasch@gmail.com>:
> >>> >> > >
> >>> >> > >>
> >>> >> > >>
> >>> >> > >> ---------- Forwarded message ---------
> >>> >> > >> От: Ivan Daschinsky <ivandasch@gmail.com>
> >>> >> > >> Date: чт, 19 нояб. 2020 г. в 13:02
> >>> >> > >> Subject: Re: IEP-61 Technical discussion
> >>> >> > >> To: Alexey Goncharuk <alexey.goncharuk@gmail.com>
> >>> >> > >>
> >>> >> > >>
> >>> >> > >> Alexey, let's arise another question. Specifically,
how nodes
> >>> >> initially
> >>> >> > >> find each other (discovery) and how they detect failures.
> >>> >> > >>
> >>> >> > >> I suppose, that gossip protocol is an ideal candidate.
For
> >>> example,
> >>> >> > >> consul [1] uses this approach, using serf [2] library
to
> discover
> >>> >> > members
> >>> >> > >> of cluster.
> >>> >> > >> Then consul forms raft ensemble (server nodes) and
client use
> >>> raft
> >>> >> > >> ensemble only as lock service.
> >>> >> > >>
> >>> >> > >> PacificA suggests internal heartbeats mechanism for
failure
> >>> >> detection of
> >>> >> > >> replicated group, but it says nothing about initial
discovery
> of
> >>> >> nodes.
> >>> >> > >>
> >>> >> > >> WDYT?
> >>> >> > >>
> >>> >> > >> [1] -- https://www.consul.io/docs/architecture/gossip
> >>> >> > >> [2] -- https://www.serf.io/
> >>> >> > >>
> >>> >> > >> чт, 19 нояб. 2020 г. в 12:46, Alexey Goncharuk
<
> >>> >> > >> alexey.goncharuk@gmail.com>:
> >>> >> > >>
> >>> >> > >>> Following up the Ignite 3.0 scope/development
approach
> threads,
> >>> >> this is
> >>> >> > >>> a separate thread to discuss technical aspects
of the IEP.
> >>> >> > >>>
> >>> >> > >>> Let's reiterate one more time on the questions
raised by Ivan
> >>> and
> >>> >> also
> >>> >> > >>> see if there are any other thoughts on the IEP:
> >>> >> > >>>
> >>> >> > >>>    - *Whether to deploy metastorage on a separate
subset of
> the
> >>> >> nodes
> >>> >> > >>>    or allow Ignite to choose these nodes automatically.*
I
> >>> think it
> >>> >> is
> >>> >> > >>>    feasible to maintain both modes: by default,
Ignite will
> >>> choose
> >>> >> > >>>    metastorage nodes automatically which essentially
will
> >>> provide
> >>> >> the
> >>> >> > same
> >>> >> > >>>    seamless user experience as TCP discovery
SPI - no separate
> >>> >> roles,
> >>> >> > >>>    simplistic deployment. For deployments where
people want to
> >>> have
> >>> >> > more
> >>> >> > >>>    fine-grained control over the nodes' assignments,
we will
> >>> >> provide a
> >>> >> > runtime
> >>> >> > >>>    configuration which will allow pinning metastorage
group to
> >>> >> certain
> >>> >> > nodes,
> >>> >> > >>>    thus eliminating the latency concerns.
> >>> >> > >>>    - *Whether there are any TLA+ specs for the
PacificA
> >>> protocol.*
> >>> >> Not
> >>> >> > >>>    to my knowledge, but it is known to be used
in production
> by
> >>> >> > Microsoft and
> >>> >> > >>>    other projects, e.g. [1]
> >>> >> > >>>
> >>> >> > >>> I would like to collect general feedback on the
IEP, as well
> as
> >>> >> > feedback
> >>> >> > >>> on specific parts of it, such as:
> >>> >> > >>>
> >>> >> > >>>    - Metastorage API
> >>> >> > >>>    - Any existing library that can be used to
avoid
> >>> re-implementing
> >>> >> the
> >>> >> > >>>    protocol ourselves? Perhaps, porting the existing
> >>> implementation
> >>> >> to
> >>> >> > Java
> >>> >> > >>>    (the way TiKV did with etcd-raft [2] [3]?
This is a very
> >>> neat way
> >>> >> > btw in my
> >>> >> > >>>    opinion because I like the finite automata-like
approach of
> >>> the
> >>> >> > replication
> >>> >> > >>>    module, and, additionally, we could sync bug
fixes and
> >>> >> improvements
> >>> >> > from
> >>> >> > >>>    the upstream project)
> >>> >> > >>>
> >>> >> > >>>
> >>> >> > >>> Thanks,
> >>> >> > >>> --AG
> >>> >> > >>>
> >>> >> > >>> [1]
> >>> >> > >>>
> >>> >>
> https://cwiki.apache.org/confluence/display/INCUBATOR/PegasusProposal
> >>> >> > >>> [2] https://github.com/etcd-io/etcd/tree/master/raft
> >>> >> > >>> [3] https://github.com/tikv/raft-rs
> >>> >> > >>>
> >>> >> > >>
> >>> >> > >>
> >>> >> > >> --
> >>> >> > >> Sincerely yours, Ivan Daschinskiy
> >>> >> > >>
> >>> >> > >>
> >>> >> > >> --
> >>> >> > >> Sincerely yours, Ivan Daschinskiy
> >>> >> > >>
> >>> >> > >
> >>> >> > >
> >>> >> > > --
> >>> >> > > Sincerely yours, Ivan Daschinskiy
> >>> >> > >
> >>> >> >
> >>> >> >
> >>> >> > --
> >>> >> > Sincerely yours, Ivan Daschinskiy
> >>> >> >
> >>> >>
> >>> >
> >>> >
> >>> > --
> >>> > Sincerely yours, Ivan Daschinskiy
> >>> >
> >>>
> >>>
> >>> --
> >>> Sincerely yours, Ivan Daschinskiy
> >>>
> >>
>


-- 
Sincerely yours, Ivan Daschinskiy

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message