spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pranay Tonpay <pranay.ton...@impetus.co.in>
Subject Re: spark-submit issue
Date Tue, 01 Sep 2015 05:14:22 GMT
Akhil,, i am not sure if that would let me catch errors, if any ? my requirement is that i
have to keep track of any error that happens and notify the user of it... i have not used
JobServer, but would take a look...


Igor,,

I changed log4j in Spark and rather than the default System.err, pointed logs to another file
on the file system and the issue got resolved for me...Looks like inline with the comment
below... ( syserr or stdout buffer being full can cause blockage )

________________________________
From: Akhil Das <akhil@sigmoidanalytics.com>
Sent: Monday, August 31, 2015 9:18 PM
To: Pranay Tonpay
Cc: Igor Berman; user@spark.apache.org
Subject: Re: spark-submit issue

Can you not use the spark jobserver instead? Just submit your job to the job server who already
has the sparkcontext initialized in it, it would make it much easier i think.

Thanks
Best Regards

On Mon, Aug 31, 2015 at 2:16 PM, Pranay Tonpay <pranay.tonpay@impetus.co.in<mailto:pranay.tonpay@impetus.co.in>>
wrote:

thx for the inputs Igor,, i am actually building an Analytics layer ( 'As a service model'
using Spark as the backend engine ) and hence i am implementing it this way...  Initially,
i was opening the spark-contenxt in the JVM that i had spawned ( without even using Spark-submit
) and adding all the jars etc in it ( as you have described )... It worked like a charm, when
i used Spark cluster as the master, but then when i tried to pass in "yarn-client" instead,
it started giving some issues ( hung jobs etc etc )... On reading some comments on forum,
i read that we should use spark-submit and so i changed my approach...  let me check more
on the 2nd point and see...


thx again for your prompt responses... BTW, do you see any issues in invoking spark-submit
in the fashion i am attempting ? ( via Runtime-exec thing as i described )


thx

pranay



________________________________
From: Igor Berman <igor.berman@gmail.com<mailto:igor.berman@gmail.com>>
Sent: Monday, August 31, 2015 1:39 PM
To: Pranay Tonpay
Cc: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: spark-submit issue

1. think once again if you want to call spark submit in such way...I'm not sure why you do
it, but please consider just opening spark context inside your jvm(you need to add spark jars
to classpath..)
2. use https://commons.apache.org/proper/commons-exec/ with PumpStreamHandler

On 31 August 2015 at 10:42, Pranay Tonpay <pranay.tonpay@impetus.co.in<mailto:pranay.tonpay@impetus.co.in>>
wrote:

Igor,, this seems to be the cause, however i am not sure at the moment how to resolve it ...
what i tried just now was that after "

SparkSubmitDriverBootstrapper" process reaches the hung stage... i went inside /proc/<pid>/fd
.... and just tailed "2" (stderr) and the process immediately exits .



From: Igor Berman <igor.berman@gmail.com<mailto:igor.berman@gmail.com>>
Sent: Monday, August 31, 2015 12:41 PM
To: Pranay Tonpay
Cc: user
Subject: Re: spark-submit issue

might be you need to drain stdout/stderr of subprocess...otherwise subprocess can deadlock
http://stackoverflow.com/questions/3054531/correct-usage-of-processbuilder

On 27 August 2015 at 16:11, pranay <pranay.tonpay@impetus.co.in<mailto:pranay.tonpay@impetus.co.in>>
wrote:
I have a java program that does this - (using Spark 1.3.1 ) Create a command
string that uses "spark-submit" in it ( with my Class file etc ), and i
store this string in a temp file somewhere as a shell script Using
Runtime.exec, i execute this script and wait for its completion, using
process.waitFor Doing ps -ef shows me SparkSubmitDriverBootstrapper , the
script running my class ... parent child relationship..

The job gets triggered on spark-cluster and gets over but
SparkSubmitDriverBootstrapper still shows up, due to this the
process.waitFor never comes out and i can't detect the execution end...

If i run the /temp file independently. things work file... only when i
trigger /temp scrict inside Runtime.exec , this issue occurs... Any comments
?



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-submit-issue-tp24474.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org<mailto:user-unsubscribe@spark.apache.org>
For additional commands, e-mail: user-help@spark.apache.org<mailto:user-help@spark.apache.org>



________________________________






NOTE: This message may contain information that is confidential, proprietary, privileged or
otherwise protected by law. The message is intended solely for the named addressee. If received
in error, please destroy and notify the sender. Any use of this email is prohibited when received
in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this
communication has been maintained nor that the communication is free of errors, virus, interception
or interference.


________________________________






NOTE: This message may contain information that is confidential, proprietary, privileged or
otherwise protected by law. The message is intended solely for the named addressee. If received
in error, please destroy and notify the sender. Any use of this email is prohibited when received
in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this
communication has been maintained nor that the communication is free of errors, virus, interception
or interference.


________________________________






NOTE: This message may contain information that is confidential, proprietary, privileged or
otherwise protected by law. The message is intended solely for the named addressee. If received
in error, please destroy and notify the sender. Any use of this email is prohibited when received
in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this
communication has been maintained nor that the communication is free of errors, virus, interception
or interference.

Mime
View raw message