spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Winandy <jonathan.wina...@gmail.com>
Subject Re: Spark madness
Date Mon, 22 May 2017 08:08:05 GMT
Hi Saikat,

You may use the wrong mailing list for your question (=> spark user).

If you want to make a single string, it's :
red.collect.mkString("\n")

Be careful of driver explosion !

Cheers,
Jonathan

On Fri, 19 May 2017, 05:21 Saikat Kanjilal, <sxk1969@hotmail.com> wrote:

> One additional point, the following line:
>
> rdd.collect.foreach(t=>println(t._2))
>
> when set to a scala string prints nothing even when I use toString at the
> end.  This seems to not be something that should be that out of the
> ordinary but I could be wrong.
>
> ------------------------------
> *From:* Saikat Kanjilal <sxk1969@hotmail.com>
> *Sent:* Thursday, May 18, 2017 8:18 PM
> *To:* dev@spark.apache.org
> *Subject:* Spark madness
>
>
> Hi Devs,
>
> I'm needing to read a json file from hdfs and turn that into a scala
> string, I have dug around for documentation on how to do this  and found
> this:
>
>
>
> http://stackoverflow.com/questions/30445263/how-to-read-whole-file-in-one-string
>
> <http://stackoverflow.com/questions/30445263/how-to-read-whole-file-in-one-string>
> How to read whole file in one string - Stack Overflow
> <http://stackoverflow.com/questions/30445263/how-to-read-whole-file-in-one-string>
> stackoverflow.com
> "How to read whole [HDFS] file in one string [in Spark, to use as sql]":
> e.g. // Put file to hdfs from edge-node's shell... hdfs dfs -put
>
>
> The following two lines of code dont seem to do the job:
>
>
> rdd = sc.wholeTextFiles("hdfs://nameservice1/user/me/test.txt")
> rdd.collect.foreach(t=>println(t._2))
>
>
> I have tried to set the second line to a scala string but that doesn't
> seem to work, I would really appreciate some insights into how to do this.
>
>
> Thanks in advance.
>
> <http://stackoverflow.com/questions/30445263/how-to-read-whole-file-in-one-string>
> How to read whole file in one string - Stack Overflow
> <http://stackoverflow.com/questions/30445263/how-to-read-whole-file-in-one-string>
> stackoverflow.com
> "How to read whole [HDFS] file in one string [in Spark, to use as sql]":
> e.g. // Put file to hdfs from edge-node's shell... hdfs dfs -put
>
>
>
>
>

Mime
View raw message