ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nikolay Izhikov <nizhi...@apache.org>
Subject Re: ExecutorServices hide assertions without logging and node stop
Date Wed, 06 May 2020 07:28:04 GMT
Hello, Maxim.

I can confirm this itching issue.
It also happens when some custom Security plugin throws an exception while processing a communication
message.

```
UUID newSecSubjId = secSubjId != null ? secSubjId : nodeId;

try (OperationSecurityContext s = ctx.security().withContext(newSecSubjId)) {
    lsnr.onMessage(nodeId, msg, plc);
}
finally {
    if (change)
        CUR_PLC.set(oldPlc);
}
```

If an exception thrown from `withContext` it is hidden from the user.

> 4 мая 2020 г., в 19:15, Maxim Muzafarov <mmuzaf@apache.org> написал(а):
> 
> Igniters,
> 
> 
> I've spent a couple of days in debug mode and found that some of the
> configured ExecutorServices of an Ignite instance completely hide
> assertion errors without any logging and node killing. This may
> violate some internal guarantees and hide serious errors.
> 
> Let's consider, for instance, GridDhtPartitionsExchangeFuture [1]. It
> has three places of submitting some Runnable on system executor
> service. If an assertion error (or even any uncached exception) occurs
> in the code block below it will be swallowed without logging, exchange
> future completion or node stoping.
> 
> cctx.kernalContext().getSystemExecutorService().submit(new Runnable() {
>    @Override public void run() {
>        sendPartitions(newCrd);
>    }
> });
> 
> I've checked that these executor services and most of them configured
> to catch only OutOfMemoryError.
> 
> Should we consider catching AssertionErrors as well and treat them as
> CRITICAL_ERRORS for the Failure Handler?
> Should we log uncached errors on each of them?
> 
> 
> The most important list of executor services configured with OOM handler only:
> execSvc
> svcExecSvc
> sysExecSvc
> p2pExecSvc
> restExecSvc
> utilityCacheExecSvc
> affExecSvc
> qryExecSvc
> 
> [1] https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/preloader/GridDhtPartitionsExchangeFuture.java#L4848


Mime
View raw message