spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andy Davidson <>
Subject RDD pipe example. Is this a bug or a feature?
Date Fri, 19 Sep 2014 18:21:20 GMT

I am wrote a little java job to try and figure out how RDD pipe works.
Bellow is my test shell script. If in the script I turn on debugging I get
output. In my console. If debugging is turned off in the shell script, I do
not see anything in my console. Is this a bug or feature?

I am running the job locally on a Mac



Here is my Java

        rdd.pipe(pwd + "/src/main/bin/").collect();



# Use this shell script to figure out how spark RDD pipe() works


set -x # turns shell debugging on

#set +x # turns shell debugging off

while read x ; 


echo $x ;


Here is the output if debugging is turned on

$ !grep

grep RDDPipe

+ echo 0

+ echo 0

+ echo 2

+ echo 0

+ echo 3

+ echo 0

+ echo 0


View raw message