spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wayne Guo <>
Subject How to know that a partition is ready when using Structured Streaming
Date Thu, 17 Jan 2019 03:36:19 GMT
When using structured streaming, we use "partitionBy" api  to partition the
output data, and use the watermark based on event-time to handle delay
records, but how to tell downstream users  that a partition is ready? For
example, when to write an empty "hadoop.done" file in a paritition

Sent from:

To unsubscribe e-mail:

View raw message