ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pavel Tupitsyn <ptupit...@apache.org>
Subject Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang
Date Fri, 09 Aug 2019 16:50:36 GMT
>  the invoker thread is going to come back eventually
Invoker thread can be anything, it can be busy, waiting or aborted.
Async is just syntactic sugar to solve "callback hell", compiler transforms
it into a state machine with callbacks.

In general, the behavior is the same in the end - callbacks/continuations
are invoked on the future/task completion thread.
Yes, there are mechanisms that are used in .NET UI frameworks to actually
dispatch the callback to the "main" thread, but
that main thread is controlled by the framework and runs a
producer/consumer queue. But that's not possible for console apps and so on.

Back to our problem - yes, using public pool for callbacks seems to be the
most obvious step.

On Fri, Aug 9, 2019 at 7:31 PM Ilya Kasnacheev <ilya.kasnacheev@gmail.com>
wrote:

> I'm not sure, but I think that in case of async, the invoker thread is
> going to come back eventually?
>
> In case of Java futures with listeners, I think listeners should be invoked
> from public pool.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 9 авг. 2019 г. в 19:16, Pavel Tupitsyn <ptupitsyn@apache.org>:
>
> > Well, you can't just take any running thread and make it run your code
> > instead, right?
> >
> > On Fri, Aug 9, 2019 at 1:32 PM Ilya Kasnacheev <
> ilya.kasnacheev@gmail.com>
> > wrote:
> >
> > > Hello!
> > >
> > > Why can't we dispatch those callbacks in the caller thread?
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > пт, 9 авг. 2019 г. в 09:50, Pavel Tupitsyn <ptupitsyn@apache.org>:
> > >
> > > > Иван,
> > > >
> > > > The fix is to dispatch those callbacks (future listeners) to a
> > different
> > > > thread pool, not sure which one though.
> > > > If I would do a .NET-only fix, I would use the default thread pool
> (non
> > > > Ignite-specific), for Java-side there is no such thing as I
> understand.
> > > >
> > > > Yes, let's have a ticket to track the issue.
> > > >
> > > > On Fri, Aug 9, 2019 at 9:17 AM Павлухин Иван <vololo100@gmail.com>
> > > wrote:
> > > >
> > > > > Ilya, Pavel,
> > > > >
> > > > > Do we a have a proposal how to fix the root cause of the problem?
> > > > > Should we a have a ticket for it?
> > > > >
> > > > > ср, 7 авг. 2019 г. в 17:48, Ilya Kasnacheev <
> > ilya.kasnacheev@gmail.com
> > > >:
> > > > > >
> > > > > > Hello!
> > > > > >
> > > > > > I think we should definitely stop running futures out of striped
> > > pool,
> > > > > > while holding any cache logs (stripe thread counts as one).
> > > > > >
> > > > > > Regards,
> > > > > > --
> > > > > > Ilya Kasnacheev
> > > > > >
> > > > > >
> > > > > > ср, 7 авг. 2019 г. в 17:20, Pavel Tupitsyn <ptupitsyn@apache.org
> >:
> > > > > >
> > > > > > > Yes, this can be done purely on .NET side, which is an
option
> > that
> > > I
> > > > > > > consider.
> > > > > > > However, the root problem is on Java side, and I believe
that
> we
> > > > > should fix
> > > > > > > the root problem.
> > > > > > >
> > > > > > > > violate some of Ignite assumptions: that we never
run user
> code
> > > > from
> > > > > > > certain thread pools
> > > > > > > We actually do run user code from Ignite thread pools:
> > > > > > >
> > > > > > > cache.getAsync(1).listen(fut ->
> > > > > > >     System.out.println("Get operation completed [value="
+
> > > fut.get()
> > > > +
> > > > > > > ']'));
> > > > > > >
> > > > > > > `println` here is executed on the striped pool. This is
stated
> in
> > > the
> > > > > > > docs that I linked above.
> > > > > > >
> > > > > > > Users have to be aware of this and they have to be very
careful
> > > with
> > > > > > > every future listener. IMO, this is a tricky gotcha and
a bad
> > > > > > > usability.
> > > > > > >
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Aug 7, 2019 at 12:22 PM Ilya Kasnacheev <
> > > > > ilya.kasnacheev@gmail.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello!
> > > > > > > >
> > > > > > > > + dev@
> > > > > > > >
> > > > > > > > I think the current behavior, where .Net callbacks
may be run
> > > from
> > > > > > > striped
> > > > > > > > pool, violate some of Ignite assumptions: that we
never run
> > user
> > > > code
> > > > > > > from
> > > > > > > > certain thread pools (like sys-stripe) and that we
try to
> limit
> > > > > options
> > > > > > > of
> > > > > > > > running user-supplied code from our internals.
> > > > > > > >
> > > > > > > > I think that future versions of .Net integration should
> remove
> > > the
> > > > > > > ability
> > > > > > > > of async callbacks to be called from non-user threads,
even
> if
> > it
> > > > can
> > > > > > > lead
> > > > > > > > to performance degradation in some cases. I suggest
removing
> > this
> > > > > mode,
> > > > > > > if
> > > > > > > > possible, while keeping only the safe one, where internal
> > threads
> > > > > are not
> > > > > > > > waiting upon completion of user code.
> > > > > > > >
> > > > > > > > In this case my issue IGNITE-12033 could be used to
track
> this
> > > > work.
> > > > > > > >
> > > > > > > > WDYT?
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > --
> > > > > > > > Ilya Kasnacheev
> > > > > > > >
> > > > > > > >
> > > > > > > > ср, 7 авг. 2019 г. в 01:47, Pavel Tupitsyn
<
> > ptupitsyn@apache.org
> > > >:
> > > > > > > >
> > > > > > > >> Sorry guys, I've completely missed this thread,
and the
> topic
> > is
> > > > > very
> > > > > > > >> important.
> > > > > > > >>
> > > > > > > >> First, a simple fix for the given example. Add
the following
> > on
> > > > the
> > > > > > > first
> > > > > > > >> line of Main:
> > > > > > > >> SynchronizationContext.SetSynchronizationContext(new
> > > > > > > >> ThreadPoolSynchronizationContext());
> > > > > > > >>
> > > > > > > >> And put the ThreadPoolSynchronizationContext class
> somewhere:
> > > > > > > >> class ThreadPoolSynchronizationContext :
> > SynchronizationContext
> > > > > > > >> {
> > > > > > > >>     // No-op.
> > > > > > > >> }
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> Now, detailed explanation. The problem exists
forever in
> > Ignite
> > > > and
> > > > > is
> > > > > > > >> mentioned in the docs briefly [1].
> > > > > > > >> Also mentioned in .NET docs (I've updated them
a bit) [2].
> > > > > > > >>
> > > > > > > >> Breakdown:
> > > > > > > >> * Ignite (Java side) runs async callbacks (continuations)
on
> > > > system
> > > > > > > >> threads, and those threads have limitations (you
should not
> > call
> > > > > Ignite
> > > > > > > >> APIs from them in general)
> > > > > > > >> * Ignite.NET wraps async operations into native
.NET Tasks
> > > > > > > >> * Usually `await ...` call in .NET will continue
execution
> on
> > > the
> > > > > > > >> original Thread (simply put, actually it is more
complex),
> so
> > > > Ignite
> > > > > > > system
> > > > > > > >> thread issue is avoided
> > > > > > > >> * However, Console applications have no
> > > `SynchronizationContext`,
> > > > > so the
> > > > > > > >> continuation can't be dispatched to original thread,
and is
> > > > > executed on
> > > > > > > >> current (Ignite) thread
> > > > > > > >> * Setting custom SynchronizationContext fixes
the issue: all
> > > async
> > > > > > > >> continuations will be dispatched to .NET thread
pool and
> never
> > > > > executed
> > > > > > > on
> > > > > > > >> Ignite threads
> > > > > > > >>
> > > > > > > >> However, dispatching callbacks to a different
thread causes
> > > > > performance
> > > > > > > >> hit, and Ignite favors performance over usability
right now.
> > > > > > > >> So it is up to the user to configure desired behavior.
> > > > > > > >>
> > > > > > > >> Let me know if you need more details.
> > > > > > > >>
> > > > > > > >> Thanks
> > > > > > > >>
> > > > > > > >> [1] https://apacheignite.readme.io/docs/async-support
> > > > > > > >> [2]
> > > https://apacheignite-net.readme.io/docs/asynchronous-support
> > > > > > > >>
> > > > > > > >> On Thu, Aug 1, 2019 at 3:41 PM Ilya Kasnacheev
<
> > > > > > > ilya.kasnacheev@gmail.com>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >>> Hello!
> > > > > > > >>>
> > > > > > > >>> I have filed a ticket about this issue so
it won't get
> lost.
> > > > > > > >>> https://issues.apache.org/jira/browse/IGNITE-12033
> > > > > > > >>>
> > > > > > > >>> Regards,
> > > > > > > >>> --
> > > > > > > >>> Ilya Kasnacheev
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> чт, 2 мая 2019 г. в 10:53, Barney Pippin
<
> > > > > > > james.prince@uk.bnpparibas.com
> > > > > > > >>> >:
> > > > > > > >>>
> > > > > > > >>>> Thanks for the response Ilya. Did you
get a chance to look
> > at
> > > > this
> > > > > > > >>>> Pavel?
> > > > > > > >>>> Thanks.
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>> --
> > > > > > > >>>> Sent from:
> http://apache-ignite-users.70518.x6.nabble.com/
> > > > > > > >>>>
> > > > > > > >>>
> > > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > > Ivan Pavlukhin
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message