spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ramandeep Singh Nanda <>
Subject Is Dataframe write blocking?
Date Thu, 08 Nov 2018 15:38:11 GMT

I have some futures setup to operate in stages, where I expect one stage to
complete before another begins. I was hoping that dataframe write call is
blocking, whereas the behavior i see is that the call returns before data
is persisted. This can cause unintended consequences. I am also using fair
scheduler so that independent jobs can run in parallel.

Spark 2.3.1.

Ramandeep Singh

View raw message