spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sen, Ranjan [USA]" <Sen_Ran...@bah.com>
Subject Re: [External] Re: no stdout output from worker
Date Mon, 10 Mar 2014 06:33:20 GMT
Hi Patrick

How do I know which part of the code is in the driver and which in task?
The structure of my code is as below-

Š

Static boolean done=false;
Š

Public static void main(..

..

JavaRDD<String> lines = ..

..

While (!done) {

..
While (..) {

JavaPairRDD<Integer, List<Integer>> labs1 = labs.map (new PairFunction<Š );

!! Here I have System.out.println (A)

} // inner while

!! Here I have System.out.println (B)


If (Š) {
	Done = true;

	!! Also here some System.out.println  (C)
	
	Break;
}

Else {

	If (Š) {
		
		!! More System.out.println  (D)

		
		labs = labs.map(Š) ;

		}
	}

} // outer while

!! Even more System.out.println  (E)

} // main 

} //class

I get the console outputs on the master for (B) and (E). I do not see any
stdout in the worker node. I find the stdout and stderr in the
<spark>/work/<appid>/0/. I see output
in stderr but not in stdout.

I do get all the outputs on the console when I run it in local mode.

Sorry I am new and may be asking some naïve question but it is really
confusing to me. Thanks for your help.

Ranjan 

On 3/9/14, 10:50 PM, "Patrick Wendell" <pwendell@gmail.com> wrote:

>Hey Sen,
>
>Is your code in the driver code or inside one of the tasks?
>
>If it's in the tasks, the place you would expect these to be is in
>stdout file under <spark>/<appid>/work/[stdout/stderr]. Are you seeing
>at least stderr logs in that folder? If not then the tasks might not
>be running on the workers machines. If you see stderr but not stdout
>that's a bit of a puzzler since they both go through the same
>mechanism.
>
>- Patrick
>
>On Sun, Mar 9, 2014 at 2:32 PM, Sen, Ranjan [USA] <Sen_Ranjan@bah.com>
>wrote:
>> Hi
>> I have some System.out.println in my Java code that is working ok in a
>>local
>> environment. But when I run the same code on a standalone  mode in a EC2
>> cluster I do not see them at the worker stdout (in the worker node under
>> <spark location>/work ) or at the driver console. Could you help me
>> understand how do I troubleshoot?
>>
>> Thanks
>> Ranjan


Mime
View raw message