tinkerpop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brett Peavler (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (TINKERPOP-2090) After running backend for a day or so System.IO.IOException keep throwing
Date Thu, 11 Apr 2019 20:39:00 GMT

    [ https://issues.apache.org/jira/browse/TINKERPOP-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815766#comment-16815766
] 

Brett Peavler edited comment on TINKERPOP-2090 at 4/11/19 8:38 PM:
-------------------------------------------------------------------

I was going through the latest ConnectionPool class code on master and I think I might have
found the cause of the issue [~s4ber7] is describing: 
{code:java}
private async Task EnsurePoolIsPopulatedAsync()
{
    // The pool could have been (partially) empty because of connection problems. So, we need
to populate it again.

    //This comparison should actually be <=
    if (_poolSize >= NrConnections) return;

    var poolState = Interlocked.CompareExchange(ref _poolState, PoolPopulationInProgress,
PoolIdle);
    if (poolState == PoolPopulationInProgress) return;
    try
    {
        await PopulatePoolAsync().ConfigureAwait(false);
    }
    finally
    {
        // We need to remove the PoolPopulationInProgress flag again even if an exception
occurred, so we don't block the pool population for ever
        Interlocked.CompareExchange(ref _poolState, PoolIdle, PoolPopulationInProgress);
    }
}
{code}
The only other place PopulatePoolAsync() is called outside of the above method is in the constructor
of the ConnectionPool class. The _poolSize is a readonly int assigned in the constructor
that will always be greater than or equal to NrConnections.

Looks like constructor is creating the initial pool of connections, the connections are being
properly disposed of when they're closed, but they're never replaced because the existing
comparison causes the method to always early exit. Eventually 0 connections are left in the
pool and the exception described is thrown.


was (Author: bpeav):
I was going through the latest ConnectionPool class code on master and I think I might have
found the cause of the issue [~s4ber7] is describing: 
{code:java}
private async Task EnsurePoolIsPopulatedAsync()
{
    // The pool could have been (partially) empty because of connection problems. So, we need
to populate it again.

    //This comparison should actually be <
    if (_poolSize >= NrConnections) return;

    var poolState = Interlocked.CompareExchange(ref _poolState, PoolPopulationInProgress,
PoolIdle);
    if (poolState == PoolPopulationInProgress) return;
    try
    {
        await PopulatePoolAsync().ConfigureAwait(false);
    }
    finally
    {
        // We need to remove the PoolPopulationInProgress flag again even if an exception
occurred, so we don't block the pool population for ever
        Interlocked.CompareExchange(ref _poolState, PoolIdle, PoolPopulationInProgress);
    }
}
{code}
The only other place PopulatePoolAsync() is called outside of the above method is in the constructor
of the ConnectionPool class. The _poolSize is a readonly int assigned in the constructor
that will always be greater than or equal to NrConnections.

Looks like constructor is creating the initial pool of connections, the connections are being
properly disposed of when they're closed, but they're never replaced because the existing
comparison causes the method to always early exit. Eventually 0 connections are left in the
pool and the exception described is thrown.

> After running backend for a day or so System.IO.IOException keep throwing
> -------------------------------------------------------------------------
>
>                 Key: TINKERPOP-2090
>                 URL: https://issues.apache.org/jira/browse/TINKERPOP-2090
>             Project: TinkerPop
>          Issue Type: Bug
>          Components: dotnet
>    Affects Versions: 3.4.0
>         Environment: .NET Core 2.1.5
> Microsoft Azure
>            Reporter: Saber Karmous
>            Priority: Critical
>
> .NET Core 2.1.5
> Gremlin.NET 3.4.0-rc2 
> We're using the latest RC of the Gremlin client. And we have a gremlin client that's
being injected as a singleton through out IoC container. After running the backend for a day
or two it keeps throwing System.IO.IOExceptions. If we restart the application it works again.
> We use Polly for out retry strategy, and retrying for 9 times. But it keeps failing.
> I added the stack trace below. Reproducing is a bit of a pain in the behind, you have
to wait for a day or two for the exception to occur.
> {noformat}
> *no* System.IO.IOException:
>  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable+ConfiguredValueTaskAwaiter.GetResult
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at System.Net.Security.SslStreamInternal+<<WriteSingleChunk>g__CompleteAsync|36_1>d`1.MoveNext
(System.Net.Security, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
>  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable+ConfiguredValueTaskAwaiter.GetResult
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at System.Net.Security.SslStreamInternal+<<WriteAsyncInternal>g__ExitWriteAsync|35_0>d`1.MoveNext
(System.Net.Security, Version=4.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
>  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at Gremlin.Net.Driver.WebSocketConnection+<SendMessageAsync>d__7.MoveNext (Gremlin.Net,
Version=3.4.0.0, Culture=neutral, PublicKeyToken=d2035e9aa387a711)
>  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at Gremlin.Net.Driver.Connection+<SendAsync>d__13.MoveNext (Gremlin.Net, Version=3.4.0.0,
Culture=neutral, PublicKeyToken=d2035e9aa387a711)
>  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at Gremlin.Net.Driver.Connection+<SubmitAsync>d__8`1.MoveNext (Gremlin.Net, Version=3.4.0.0,
Culture=neutral, PublicKeyToken=d2035e9aa387a711)
>  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at Gremlin.Net.Driver.ProxyConnection+<SubmitAsync>d__3`1.MoveNext (Gremlin.Net,
Version=3.4.0.0, Culture=neutral, PublicKeyToken=d2035e9aa387a711)
>  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at Gremlin.Net.Driver.GremlinClient+<SubmitAsync>d__6`1.MoveNext (Gremlin.Net,
Version=3.4.0.0, Culture=neutral, PublicKeyToken=d2035e9aa387a711)
>  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at Gremlin.Net.Driver.GremlinClientExtensions+<SubmitAsync>d__4`1.MoveNext (Gremlin.Net,
Version=3.4.0.0, Culture=neutral, PublicKeyToken=d2035e9aa387a711)
>  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at boskalis.world.data.repository.Services.AzureCosmosDBGremlinProvider+<>c__DisplayClass9_0`1+<<Query>b__5>d.MoveNext
(boskalis.world.data.logic, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullboskalis.world.data.logic,
Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\boskalis.world.data.logic\Services\AzureCosmosDBGremlinProvider.csboskalis.world.data.logic,
Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 222)
>  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at Polly.Policy+<>c__DisplayClass181_0`1+<<ExecuteAsyncInternal>b__0>d.MoveNext
(Polly, Version=6.0.0.0, Culture=neutral, PublicKeyToken=c8a3ffc3f8f825cc)
>  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
(System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at Polly.RetrySyntaxAsync+<>c__DisplayClass25_1+<<WaitAndRetryAsync>b__1>d.MoveNext
(Polly, Version=6.0.0.0, Culture=neutral, PublicKeyToken=c8a3ffc3f8f825cc)
>  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
>  at Polly.Retry.RetryEngine+<ImplementationAsync>d__1`1.MoveNext (Polly, Version=6.0.0.0,
Culture=neutral, PublicKeyToken=c8a3ffc3f8f825cc)
> Inner exception System.Net.Sockets.SocketException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
> {noformat}
> I'm going to implement a workaround by recycling the client every couple of hours.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message