spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mehdi Singer <mehdi.sin...@lampiris.be>
Subject RE: Problem executing Spark via JBoss application
Date Thu, 16 Oct 2014 06:49:46 GMT
Indeed it was a problem on the executor side… I have to figure out how to fix it now ;-)

Thanks!

Mehdi

De : Yana Kadiyska [mailto:yana.kadiyska@gmail.com]
Envoyé : mercredi 15 octobre 2014 18:32
À : Mehdi Singer
Cc : user@spark.apache.org
Objet : Re: Problem executing Spark via JBoss application

From this line :  Removing executor app-20141015142644-0125/0 because it is EXITED I would
guess that you need to examine the executor log to see why the executor actually exited. My
guess would be that the executor cannot connect back to your driver. But check the log from
the executor. It should be in SPARK_HOME/work/app-id/executor_id/stderr on the worker box,
I believe.

On Wed, Oct 15, 2014 at 8:56 AM, Mehdi Singer <mehdi.singer@lampiris.be<mailto:mehdi.singer@lampiris.be>>
wrote:
Hi,

I have a Spark standalone example application which is working fine.
I'm now trying to integrate this application into a J2EE application, deployed on JBoss 7.1.1
and accessed via a web service. The JBoss server is installed on my local machine (Windows
7) and the master Spark is remote (Linux).
The example simply executes a count on my RDD.
When I call the webservice I'm getting the following error at JBoss side when executing the
count:

11:48:10,232 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/el2-etrm-spark].[ws]]
(http--127.0.0.1-8082-3) "Servlet.service()" pour la servlet ws a généré une exception:
java.lang.RuntimeException: org.apache.cxf.interceptor.Fault: Job cancelled because SparkContext
was shut down
                at org.apache.cxf.interceptor.AbstractFaultChainInitiatorObserver.onMessage(AbstractFaultChainInitiatorObserver.java:116)
[cxf-api-2.6.9.jar:2.6.9]
                at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:322)
[cxf-api-2.6.9.jar:2.4.3]
                at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
[cxf-api-2.6.9.jar:2.6.9]
                at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:211)
[cxf-bundle-2.6.2.jar:2.6.2]
                at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:213)
[cxf-bundle-2.6.2.jar:2.6.2]
                at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:154)
[cxf-bundle-2.6.2.jar:2.6.2]
                at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:130)
[cxf-bundle-2.6.2.jar:2.6.2]
                at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:221)
[cxf-bundle-2.6.2.jar:2.6.2]
                at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:146)
[cxf-bundle-2.6.2.jar:2.6.2]
                at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]
                at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:197)
[cxf-bundle-2.6.2.jar:2.6.2]
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329)
[jbossweb-7.0.13.Final.jar:]
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
[jbossweb-7.0.13.Final.jar:]
                at org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter.doFilterInternal(OpenEntityManagerInViewFilter.java:180)
[spring-orm-3.2.3.RELEASE.jar:3.2.3.RELEASE]
                at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
[spring-web-3.2.3.RELEASE.jar:3.2.3.RELEASE]
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
[jbossweb-7.0.13.Final.jar:]
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
[jbossweb-7.0.13.Final.jar:]
                at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:186)
[spring-security-web-3.1.3.RELEASE.jar:3.1.3.RELEASE]
                at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
[spring-security-web-3.1.3.RELEASE.jar:3.1.3.RELEASE]
                at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
[spring-web-3.2.3.RELEASE.jar:3.2.3.RELEASE]
                at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)
[spring-web-3.2.3.RELEASE.jar:3.2.3.RELEASE]
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
[jbossweb-7.0.13.Final.jar:]
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
[jbossweb-7.0.13.Final.jar:]
                at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
[jbossweb-7.0.13.Final.jar:]
                at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
[jbossweb-7.0.13.Final.jar:]
                at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50)
[jboss-as-jpa-7.1.1.Final.jar:7.1.1.Final]
                at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153)
[jboss-as-web-7.1.1.Final.jar:7.1.1.Final]
                at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
[jbossweb-7.0.13.Final.jar:]
                at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
[jbossweb-7.0.13.Final.jar:]
                at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
[jbossweb-7.0.13.Final.jar:]
                at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)
[jbossweb-7.0.13.Final.jar:]
                at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877)
[jbossweb-7.0.13.Final.jar:]
                at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671)
[jbossweb-7.0.13.Final.jar:]
                at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930)
[jbossweb-7.0.13.Final.jar:]
                at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
Caused by: org.apache.cxf.interceptor.Fault: Job cancelled because SparkContext was shut down
                at org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:162)
[cxf-api-2.6.9.jar:2.6.9]
                at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:128)
[cxf-api-2.6.9.jar:2.6.9]
                at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:167) [cxf-bundle-2.6.2.jar:2.6.2]
                at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:94) [cxf-bundle-2.6.2.jar:2.6.2]
                at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:57)
[cxf-api-2.6.9.jar:2.6.9]
                at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:93)
[cxf-api-2.6.9.jar:2.6.9]
                at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:262)
[cxf-api-2.6.9.jar:2.4.3]
                ... 33 more
Caused by: org.apache.spark.SparkException: Job cancelled because SparkContext was shut down
                at org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:694)
[spark-core_2.10-1.1.0.jar:1.1.0]
                at org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:693)
[spark-core_2.10-1.1.0.jar:1.1.0]
                at scala.collection.mutable.HashSet.foreach(HashSet.scala:79) [scala-library-2.10.4.jar:]
                at org.apache.spark.scheduler.DAGScheduler.cleanUpAfterSchedulerStop(DAGScheduler.scala:693)
[spark-core_2.10-1.1.0.jar:1.1.0]
                at org.apache.spark.scheduler.DAGSchedulerEventProcessActor.postStop(DAGScheduler.scala:1399)
[spark-core_2.10-1.1.0.jar:1.1.0]
                at akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:201)
[akka-actor_2.10-2.2.3-shaded-protobuf.jar:]
                at akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:163)
[akka-actor_2.10-2.2.3-shaded-protobuf.jar:]
                at akka.actor.ActorCell.terminate(ActorCell.scala:338) [akka-actor_2.10-2.2.3-shaded-protobuf.jar:]
                at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:431) [akka-actor_2.10-2.2.3-shaded-protobuf.jar:]
                at akka.actor.ActorCell.systemInvoke(ActorCell.scala:447) [akka-actor_2.10-2.2.3-shaded-protobuf.jar:]
                at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:262) [akka-actor_2.10-2.2.3-shaded-protobuf.jar:]
                at akka.dispatch.Mailbox.run(Mailbox.scala:218) [akka-actor_2.10-2.2.3-shaded-protobuf.jar:]
                at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
[akka-actor_2.10-2.2.3-shaded-protobuf.jar:]
                at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [scala-library-2.10.4.jar:]
                at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
[scala-library-2.10.4.jar:]
                at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
[scala-library-2.10.4.jar:]
                at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[scala-library-2.10.4.jar:]

On Spark master side I'm getting the following output:

14/10/15 11:48:10 INFO Master: akka.tcp://sparkDriver@mymachinename:51834 got disassociated,
removing it.
14/10/15 11:48:10 INFO Master: akka.tcp://sparkDriver@mymachinename:51834 got disassociated,
removing it.
14/10/15 11:48:10 INFO LocalActorRef: Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying]
from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.21.6.4%3A51850-649#254460547]
was not delivered. [215] dead letters encountered. This logging can be turned off or adjusted
with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14/10/15 11:48:10 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@sparkmastermachinename:7077]
-> [akka.tcp://sparkDriver@mymachinename:51834]: Error [Association failed with [akka.tcp://sparkDriver@mymachinename:51834]]
[
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkDriver@mymachinename:51834]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection
refused: mymachinename/10.21.6.4:51834<http://10.21.6.4:51834>
]
14/10/15 11:48:10 INFO Master: akka.tcp://sparkDriver@mymachinename:51834 got disassociated,
removing it.
14/10/15 11:48:10 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@sparkmastermachinename:7077]
-> [akka.tcp://sparkDriver@mymachinename:51834]: Error [Association failed with [akka.tcp://sparkDriver@mymachinename:51834]]
[
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkDriver@mymachinename:51834]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection
refused: mymachinename/10.21.6.4:51834<http://10.21.6.4:51834>
]
14/10/15 11:48:10 INFO Master: akka.tcp://sparkDriver@mymachinename:51834 got disassociated,
removing it.
14/10/15 11:48:10 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@sparkmastermachinename:7077]
-> [akka.tcp://sparkDriver@mymachinename:51834]: Error [Association failed with [akka.tcp://sparkDriver@mymachinename:51834]]
[
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkDriver@mymachinename:51834]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection
refused: mymachinename/10.21.6.4:51834<http://10.21.6.4:51834>
]
14/10/15 11:48:10 INFO Master: akka.tcp://sparkDriver@mymachinename:51834 got disassociated,
removing it.

I've excluded the "Connection refused" messages from the possible reasons for my problem as
I have the same message when I execute my example as standalone application...

For information, here's the Spark master output when I execute the example in standalone:

14/10/15 14:26:44 INFO Master: Registering app ts-aggregation
14/10/15 14:26:44 INFO Master: Registered app ts-aggregation with ID app-20141015142644-0125
14/10/15 14:26:44 INFO Master: Launching executor app-20141015142644-0125/0 on worker worker-20141015100053-sparkmastermachinename.lampiris.biz-39058
14/10/15 14:26:44 INFO Master: Launching executor app-20141015142644-0125/1 on worker worker-20141015100053-sparkmastermachinename.lampiris.biz-46115
14/10/15 14:26:48 INFO Master: Removing executor app-20141015142644-0125/0 because it is EXITED
14/10/15 14:26:48 INFO Master: Removing executor app-20141015142644-0125/1 because it is EXITED
14/10/15 14:26:49 INFO Master: akka.tcp://sparkDriver@mymachinename:58525 got disassociated,
removing it.
14/10/15 14:26:49 INFO Master: Removing app app-20141015142644-0125
14/10/15 14:26:49 INFO LocalActorRef: Message [akka.remote.transport.AssociationHandle$Disassociated]
from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.21.6.4%3A58543-653#1905465071]
was not delivered. [216] dead letters encountered. This logging can be turned off or adjusted
with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14/10/15 14:26:49 INFO LocalActorRef: Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying]
from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.21.6.4%3A58543-653#1905465071]
was not delivered. [217] dead letters encountered. This logging can be turned off or adjusted
with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14/10/15 14:26:49 INFO Master: akka.tcp://sparkDriver@mymachinename:58525 got disassociated,
removing it.
14/10/15 14:26:49 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@sparkmastermachinename:7077]
-> [akka.tcp://sparkDriver@mymachinename:58525]: Error [Association failed with [akka.tcp://sparkDriver@mymachinename:58525]]
[
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkDriver@mymachinename:58525]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection
refused: mymachinename/10.21.6.4:58525<http://10.21.6.4:58525>
]
14/10/15 14:26:49 INFO Master: akka.tcp://sparkDriver@mymachinename:58525 got disassociated,
removing it.
14/10/15 14:26:49 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@sparkmastermachinename:7077]
-> [akka.tcp://sparkDriver@mymachinename:58525]: Error [Association failed with [akka.tcp://sparkDriver@mymachinename:58525]]
[
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkDriver@mymachinename:58525]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection
refused: mymachinename/10.21.6.4:58525<http://10.21.6.4:58525>
]
14/10/15 14:26:49 INFO Master: akka.tcp://sparkDriver@mymachinename:58525 got disassociated,
removing it.
14/10/15 14:26:49 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@sparkmastermachinename:7077]
-> [akka.tcp://sparkDriver@mymachinename:58525]: Error [Association failed with [akka.tcp://sparkDriver@mymachinename:58525]]
[
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkDriver@mymachinename:58525]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection
refused: mymachinename/10.21.6.4:58525<http://10.21.6.4:58525>
]
14/10/15 14:26:49 INFO Master: akka.tcp://sparkDriver@mymachinename:58525 got disassociated,
removing it.

Does anyone have any clue on what may cause this error?

Thank you for your help !

Regards


Mime
View raw message