kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ewen Cheslack-Postava <e...@confluent.io>
Subject Re: kafka-connect-hdfs failure due to corrupt WAL file
Date Wed, 03 Aug 2016 05:14:39 GMT
If you're not worried about duplicates, then yes, you can delete the WAL to
recover.

By the way, we're aware of some issues in that code that we're addressing
here: https://github.com/confluentinc/kafka-connect-hdfs/pull/96

-Ewen

On Tue, Aug 2, 2016 at 3:12 PM, Prabhu V <vprabhu@gmail.com> wrote:

> Hi,
>
> I am using kafka-connect-hdfs in a 2 nodes and one of the nodes had to be
> rebooted when the process was running.
>
> Upon restart the process fails with
>
> 16/08/02 21:43:30 ERROR hdfs.TopicPartitionWriter: Recovery failed at state
> RECOVERY_PARTITION_PAUSED
> org.apache.kafka.connect.errors.ConnectException: java.io.IOException:
> Filesystem closed
>         at io.confluent.connect.hdfs.wal.FSWAL.apply(FSWAL.java:131)
>
> I looked into the code and looks like when the WAL log file may be
> corrupt/unreadeable, it causes this issue.
>
> Can I delete the WAL files to fix this issue ? I dont mind duplicate
> events.
>
> Thanks,
> Prabhu
>



-- 
Thanks,
Ewen

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message