spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: Spark Debugging
Date Thu, 30 Oct 2014 10:40:55 GMT
Thanks
Best Regards

On Thu, Oct 30, 2014 at 1:43 PM, Akhil Das <akhil@sigmoidanalytics.com>
wrote:

> Awesome.
>
> Thanks
> Best Regards
>
> On Thu, Oct 30, 2014 at 1:30 PM, Naveen Kumar Pokala <
> npokala@spcapitaliq.com> wrote:
>
>> Thanks Akhil, It is working.
>>
>>
>>
>> Regards,
>>
>> Naveen
>>
>>
>>
>> *From:* Akhil Das [mailto:akhil@sigmoidanalytics.com]
>> *Sent:* Thursday, October 30, 2014 1:24 PM
>>
>> *To:* Naveen Kumar Pokala
>> *Subject:* Re: Spark Debugging
>>
>>
>>
>> Yes you can, but the values might not give you a complete clue. Give it a
>> try
>>
>>
>> Thanks
>>
>> Best Regards
>>
>>
>>
>> On Thu, Oct 30, 2014 at 1:20 PM, Naveen Kumar Pokala <
>> npokala@spcapitaliq.com> wrote:
>>
>> Do you mean can’t I debug line by line?
>>
>>
>>
>> Thanks and Regards
>>
>> Naveen
>>
>>
>>
>> *From:* Akhil Das [mailto:akhil@sigmoidanalytics.com]
>> *Sent:* Thursday, October 30, 2014 1:19 PM
>>
>>
>> *To:* Naveen Kumar Pokala
>> *Subject:* Re: Spark Debugging
>>
>>
>>
>> Master machine. The execution will get pause in eclipse regardless of
>> where its executing the program, you might not get a complete flow, but you
>> can see whats happening.
>>
>>
>> Thanks
>>
>> Best Regards
>>
>>
>>
>> On Thu, Oct 30, 2014 at 1:10 PM, Naveen Kumar Pokala <
>> npokala@spcapitaliq.com> wrote:
>>
>> Hi Akhil,
>>
>>
>>
>> With below configuration I am successfully able to connect to  Node
>> A(Master node) where I am running the spark submit program, Spark may give
>> driver program to another Node B also.
>>
>>
>>
>> Which machine IP should I use to debug the programs from eclipse.
>>
>>
>>
>> Thanks,
>>
>> Naveen
>>
>>
>>
>> *From:* Akhil Das [mailto:akhil@sigmoidanalytics.com]
>> *Sent:* Thursday, October 30, 2014 12:35 PM
>> *To:* Naveen Kumar Pokala
>> *Subject:* Re: Spark Debugging
>>
>>
>>
>> ​Hi Naveen,
>>
>>
>>
>> You should be able to connect to 9999 port also from your windows machine.
>> ​
>>
>>
>> Thanks
>>
>> Best Regards
>>
>>
>>
>> On Thu, Oct 30, 2014 at 12:32 PM, Naveen Kumar Pokala <
>> npokala@spcapitaliq.com> wrote:
>>
>> Hi Akhil,
>>
>>
>>
>> I have gone through the article. But I am  not starting my java program
>> with Java command, I am submitting through spark-submit and I have added
>> like below is that ok?
>>
>>
>>
>> bin/spark-submit --class sample.spark.test.SparkJob --master yarn-cluster
>> --driver-memory 4g --executor-memory  --conf
>> "spark.executor.extraJavaOptions=-Xdebug
>> -agentlib:jdwp=transport=dt_socket,address=9999,server=y,suspend=n"
>> /home/npokala/data/test.jar
>>
>>
>>
>> I am able to connect to cluster from my windows machine only through
>> putty. I am able to access the cluster information and spark information
>> from my browser. Is that sufficient.?
>>
>>
>>
>>
>>
>> Regards,
>>
>> Naveen.
>>
>>
>>
>> *From:* Akhil Das [mailto:akhil@sigmoidanalytics.com]
>> *Sent:* Thursday, October 30, 2014 12:10 PM
>> *To:* Naveen Kumar Pokala
>> *Cc:* user@spark.apache.org
>> *Subject:* Re: Spark Debugging
>>
>>
>>
>> Its called remote debugging. You can read this article
>> <http://www.eclipse.org/jetty/documentation/current/enable-remote-debugging.html>
>> for more information. You will have to make sure that the network between
>> your cluster and windows machine can communicate with each other.
>>
>>
>> Thanks
>>
>> Best Regards
>>
>>
>>
>> On Thu, Oct 30, 2014 at 12:06 PM, Naveen Kumar Pokala <
>> npokala@spcapitaliq.com> wrote:
>>
>> Hi,
>>
>>
>>
>>   I have installed 2 node hadoop cluster (For example, on Unix machines A
>> and B. A master node and data node, B is data node)
>>
>>
>>
>>   I am submitting my driver programs through SPARK 1.1.0 with
>> bin/spark-submit from Putty Client from my Windows machine.
>>
>>
>>
>> I want to debug my program from Eclipse on my local machine. I am not
>> able to find a way to debug.
>>
>>
>>
>> Please let me know the ways to debug my driver program as well as
>> executor programs
>>
>>
>>
>>
>>
>> Regards,
>>
>>
>>
>> Naveen.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>

Mime
View raw message