spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chetan Khatri <>
Subject Writing to Parquet Job turns to wait mode after even completion of job
Date Fri, 21 Oct 2016 20:47:03 GMT
Hello Spark Users,

I am writing around 10 GB of Processed Data to Parquet where having 1 TB of
HDD and 102 GB of RAM, 16 vCore machine on Google Cloud.

Every time, i write to parquet. it shows on Spark UI that stages succeeded
but on spark shell it hold context on wait mode for almost 10 mins. then it
clears broadcast, accumulator shared variables.

Can we sped up this thing ?


Yours Aye,
Chetan Khatri.
M.+91 76666 80574
Data Science Researcher

​​Statement of Confidentiality
The contents of this e-mail message and any attachments are confidential
and are intended solely for addressee. The information may also be legally
privileged. This transmission is sent in trust, for the sole purpose of
delivery to the intended recipient. If you have received this transmission
in error, any use, reproduction or dissemination of this transmission is
strictly prohibited. If you are not the intended recipient, please
immediately notify the sender by reply e-mail or phone and delete this
message and its attachments, if any.​​

View raw message