spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sen, Ranjan [USA]" <>
Subject Re: [External] Re: no stdout output from worker
Date Mon, 10 Mar 2014 06:33:20 GMT
Hi Patrick

How do I know which part of the code is in the driver and which in task?
The structure of my code is as below-


Static boolean done=false;

Public static void main(..


JavaRDD<String> lines = ..


While (!done) {

While (..) {

JavaPairRDD<Integer, List<Integer>> labs1 = (new PairFunction<Š );

!! Here I have System.out.println (A)

} // inner while

!! Here I have System.out.println (B)

If (Š) {
	Done = true;

	!! Also here some System.out.println  (C)

Else {

	If (Š) {
		!! More System.out.println  (D)

		labs =Š) ;


} // outer while

!! Even more System.out.println  (E)

} // main 

} //class

I get the console outputs on the master for (B) and (E). I do not see any
stdout in the worker node. I find the stdout and stderr in the
<spark>/work/<appid>/0/. I see output
in stderr but not in stdout.

I do get all the outputs on the console when I run it in local mode.

Sorry I am new and may be asking some naïve question but it is really
confusing to me. Thanks for your help.


On 3/9/14, 10:50 PM, "Patrick Wendell" <> wrote:

>Hey Sen,
>Is your code in the driver code or inside one of the tasks?
>If it's in the tasks, the place you would expect these to be is in
>stdout file under <spark>/<appid>/work/[stdout/stderr]. Are you seeing
>at least stderr logs in that folder? If not then the tasks might not
>be running on the workers machines. If you see stderr but not stdout
>that's a bit of a puzzler since they both go through the same
>- Patrick
>On Sun, Mar 9, 2014 at 2:32 PM, Sen, Ranjan [USA] <>
>> Hi
>> I have some System.out.println in my Java code that is working ok in a
>> environment. But when I run the same code on a standalone  mode in a EC2
>> cluster I do not see them at the worker stdout (in the worker node under
>> <spark location>/work ) or at the driver console. Could you help me
>> understand how do I troubleshoot?
>> Thanks
>> Ranjan

View raw message