spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From cane <>
Subject saveAsNewAPIHadoopDataset must not enable speculation for parquet file?
Date Tue, 03 Apr 2018 10:19:40 GMT
Now, if we use saveAsNewAPIHadoopDataset with speculation enable.It may cause
data loss.
I check the comment of thi api:

  We should make sure our tasks are idempotent when speculation is enabled,
i.e. do
   * not use output committer that writes data directly.
   * There is an example in to show the bad
   * result of using direct output committer with speculation enabled.

But if this the rule we must follow?
For example,for parquet it will got ParquetOutPutCommitter.
In this case, speculation must disable for parquet?

Is there some one know the history?
Thanks too much!

Sent from:

To unsubscribe e-mail:

View raw message