crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Micah Whitacre (JIRA)" <>
Subject [jira] [Updated] (CRUNCH-621) Kafka no data to consume could get stuck
Date Tue, 13 Sep 2016 15:50:21 GMT


Micah Whitacre updated CRUNCH-621:
    Attachment: CRUNCH-621b.patch

Took a different approach where it would check after repeated empty failures because Kafka
could still drop data after checking proactively.  Also will cut down on calls to Kafka only
when it fails vs calling from each task.

(hopefully spacing nits not a problem this time)

> Kafka no data to consume could get stuck
> ----------------------------------------
>                 Key: CRUNCH-621
>                 URL:
>             Project: Crunch
>          Issue Type: Bug
>          Components: IO
>            Reporter: Micah Whitacre
>            Assignee: Micah Whitacre
>         Attachments: CRUNCH-621.patch, CRUNCH-621b.patch
> With the KafkaSource and specifically the KafkaRecordReader, if there is no data to consume
from a partition because all data fell off Kafka then we can end up in a situation where the
KafkaRecordReader will poll forever as it never gets data to reach its ending offset.  

This message was sent by Atlassian JIRA

View raw message