atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashutosh Mestry <>
Subject Re: Review Request 61665: ATLAS-2047: Exception Thrown by Kafka Consumer Ends up Filling Logs Due to Incorrect Handling
Date Thu, 17 Aug 2017 05:04:03 GMT

This is an automatically generated e-mail. To reply, visit:

(Updated Aug. 17, 2017, 5:04 a.m.)

Review request for atlas, Madhan Neethiraj and Nixon Rodrigues.


Updates include:
- Additional analysis.
- Implementation details.

Bugs: ATLAS-2047

Repository: atlas

Description (updated)

Please refer to [ATLAS-2047]( for background
and analysis.


The _IllegalStateException_ is thrown by _KafkaConsumer.aquire_. This method is called at
the beginning of almost every method in this class. The method checks if the consumer is closed,
if it is then it throws IllegalStateException.

Scenario may come about in this way:
- Shutdown has been initiated. Close on consumer is called.
- However, the consumer thread is just about to enter another poll cycle.
- Thus acquire sees that consumer is closed and throws the exception (2nd bullet above).

Please take a look at this scala code. This is _ShutdownableThread_. The thread does the job
of handling all exceptions. Upon exception, it manages the _shutdownLatch_ (from yesterday’s
bug fix) and gets out of the _isRunning_ loop.
  override def run(): Unit = {
    info("Starting ")
    } catch{
      case e: Throwable =>
          error("Error due to ", e)
    info("Stopped ")


Special treatment is given to _IllegalStateException_ by implementing pause & retry logic:
- Modified _LOG_ to _debug_. That way logs are not filled during retry.
- _HookConsumer_ is more resilient. It handles exceptions resulting from _Kafka_ and entity


  webapp/src/main/java/org/apache/atlas/notification/ ef64c3b

  webapp/src/test/java/org/apache/atlas/notification/ a6f58e8



**Unit tests**
Updated unit tests to reproduce the scenarios and verify the fix.

**Functional tests**
Verified regular notification scenarios.


Ashutosh Mestry

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message