spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jey Kottalam <...@cs.berkeley.edu>
Subject Re: RDD pipe example. Is this a bug or a feature?
Date Fri, 19 Sep 2014 19:35:32 GMT
Hi Andy,

That's a feature -- you'll have to print out the return value from
collect() if you want the contents to show up on stdout.

Probably something like this:

for(Iterator<String> iter = rdd.pipe(pwd +
"/src/main/bin/RDDPipe.sh").collect().iterator(); iter.hasNext();)
   System.out.println(iter.next());


Hope that helps,
-Jey

On Fri, Sep 19, 2014 at 11:21 AM, Andy Davidson
<Andy@santacruzintegration.com> wrote:
> Hi
>
> I am wrote a little java job to try and figure out how RDD pipe works.
> Bellow is my test shell script. If in the script I turn on debugging I get
> output. In my console. If debugging is turned off in the shell script, I do
> not see anything in my console. Is this a bug or feature?
>
> I am running the job locally on a Mac
>
> Thanks
>
> Andy
>
>
> Here is my Java
>
>         rdd.pipe(pwd + "/src/main/bin/RDDPipe.sh").collect();
>
>
>
> #!/bin/sh
>
>
> #
>
> # Use this shell script to figure out how spark RDD pipe() works
>
> #
>
>
> set -x # turns shell debugging on
>
> #set +x # turns shell debugging off
>
>
> while read x ;
>
> do
>
> echo RDDPipe.sh $x ;
>
> Done
>
>
>
> Here is the output if debugging is turned on
>
> $ !grep
>
> grep RDDPipe run.sh.out
>
> + echo RDDPipe.sh 0
>
> + echo RDDPipe.sh 0
>
> + echo RDDPipe.sh 2
>
> + echo RDDPipe.sh 0
>
> + echo RDDPipe.sh 3
>
> + echo RDDPipe.sh 0
>
> + echo RDDPipe.sh 0
>
> $

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message