spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pranay Tonpay <>
Subject Re: spark-submit issue
Date Mon, 31 Aug 2015 07:42:25 GMT
Igor,, this seems to be the cause, however i am not sure at the moment how to resolve it ...
what i tried just now was that after "

SparkSubmitDriverBootstrapper" process reaches the hung stage... i went inside /proc/<pid>/fd
.... and just tailed "2" (stderr) and the process immediately exits .

From: Igor Berman <>
Sent: Monday, August 31, 2015 12:41 PM
To: Pranay Tonpay
Cc: user
Subject: Re: spark-submit issue

might be you need to drain stdout/stderr of subprocess...otherwise subprocess can deadlock

On 27 August 2015 at 16:11, pranay <<>>
I have a java program that does this - (using Spark 1.3.1 ) Create a command
string that uses "spark-submit" in it ( with my Class file etc ), and i
store this string in a temp file somewhere as a shell script Using
Runtime.exec, i execute this script and wait for its completion, using
process.waitFor Doing ps -ef shows me SparkSubmitDriverBootstrapper , the
script running my class ... parent child relationship..

The job gets triggered on spark-cluster and gets over but
SparkSubmitDriverBootstrapper still shows up, due to this the
process.waitFor never comes out and i can't detect the execution end...

If i run the /temp file independently. things work file... only when i
trigger /temp scrict inside Runtime.exec , this issue occurs... Any comments

View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:<>
For additional commands, e-mail:<>


NOTE: This message may contain information that is confidential, proprietary, privileged or
otherwise protected by law. The message is intended solely for the named addressee. If received
in error, please destroy and notify the sender. Any use of this email is prohibited when received
in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this
communication has been maintained nor that the communication is free of errors, virus, interception
or interference.

View raw message