spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nuthan Reddy <nut...@sigmoidanalytics.com>
Subject Re: How to print DataFrame.show(100) to text file at HDFS
Date Sun, 14 Apr 2019 05:01:27 GMT
Hi Chetan,

You can use

spark-submit showDF.py | hadoop fs -put - showDF.txt

showDF.py:

from pyspark.sql import SparkSession


spark = SparkSession.builder.appName("Write stdout").getOrCreate()

spark.sparkContext.setLogLevel("OFF")


spark.table("<yourdf>").show(100,truncate=false)

But is there any specific reason you want to write it to hdfs? Is this for
human consumption?

Regards,
Nuthan

On Sat, Apr 13, 2019 at 6:41 PM Chetan Khatri <chetan.opensource@gmail.com>
wrote:

> Hello Users,
>
> In spark when I have a DataFrame and do  .show(100) the output which gets
> printed, I wants to save as it is content to txt file in HDFS.
>
> How can I do this?
>
> Thanks
>


-- 
Nuthan Reddy
Sigmoid Analytics

-- 
*Disclaimer*: This is not a mass e-mail and my intention here is purely 
from a business perspective, and not to spam or encroach your privacy. I am 
writing with a specific agenda to build a personal business connection. 
Being a reputed and genuine organization, Sigmoid respects the digital 
security of every prospect and tries to comply with GDPR and other regional 
laws. Please let us know if you feel otherwise and we will rectify the 
misunderstanding and adhere to comply in the future. In case we have missed 
any of the compliance, it is completely unintentional.

Mime
View raw message