spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ulysses you (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-28597) Spark streaming terminated when close meta data log error
Date Fri, 02 Aug 2019 01:05:00 GMT
ulysses you created SPARK-28597:
-----------------------------------

             Summary: Spark streaming terminated when close meta data log error 
                 Key: SPARK-28597
                 URL: https://issues.apache.org/jira/browse/SPARK-28597
             Project: Spark
          Issue Type: Bug
          Components: DStreams, Structured Streaming
    Affects Versions: 2.4.3
            Reporter: ulysses you


Now, HDFSMetaDataLog is the only implement of MetaDataLog. If HDFSMetaDataLog.writeBatchToFile()
execute error, then total app will be terminated.

The meta data log is very important for spark, we should retry some times when exception,
and add a config to control retry times.

For example HDFS has a bug that close error if has no enough replicas more detail HDFS-11486.

 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message