spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amit Joshi <>
Subject [SPARK-STRUCTURED-STREAMING] IllegalStateException: Race while writing batch 4
Date Fri, 07 Aug 2020 19:18:54 GMT

I have 2spark structure streaming queries writing to the same outpath in
object storage.
Once in a while I am getting the "IllegalStateException: Race while writing
batch 4".
I found that this error is because there are two writers writing to the
output path. The file streaming sink doesn't support multiple writers.
It assumes there is only one writer writing to the path. Each query needs
to use its own output directory.

Is there a way to write the output to the same path by both queries, as I
need the output at the same path.?

Amit Joshi

View raw message