spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Vanzin (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-23497) Sparklyr Applications doesn't disconnect spark driver in client mode
Date Tue, 12 Feb 2019 21:17:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-23497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Marcelo Vanzin resolved SPARK-23497.
------------------------------------
    Resolution: Invalid

See above.

> Sparklyr Applications doesn't disconnect spark driver in client mode
> --------------------------------------------------------------------
>
>                 Key: SPARK-23497
>                 URL: https://issues.apache.org/jira/browse/SPARK-23497
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core, YARN
>    Affects Versions: 2.1.0
>            Reporter: bharath kumar
>            Priority: Major
>
> Hello,
> When we use Sparklyr to connect to Yarn cluster manager in client mode or cluster mode,
Spark driver will not disconnect unless we mention the spark_disconnect(sc) in the code.
> Does it make sense to add a timeout feature for driver to exit after certain amount of
time, in client mode or cluster mode. I think its only happening with connection from Sparklyr
to Yarn. Some times the driver stays there for weeks and holds minimum resources .
> *More  Details:*
> Yarn -2.7.0
> Spark -2.1.0
> Rversion:
> Microsoft R Open 3.4.2
> Rstudio Version:
> rstudio-server-1.1.414-1.x86_64
> yarn application -status application_id
> 18/01/22 09:08:45 INFO client.MapRZKBasedRMFailoverProxyProvider: Updated RM address
to resourcemanager.com/resourcemanager:8032
>  
> Application Report : 
>         Application-Id : application_id
>         Application-Name : sparklyr
>         Application-Type : SPARK
>         User : userid
>         Queue : root.queuename
>         Start-Time : 1516245523965
>         Finish-Time : 0
>         Progress : 0%
>         State : RUNNING
>         Final-State : UNDEFINED
>         Tracking-URL : N/A
>         RPC Port : -1
>         AM Host : N/A
>         Aggregate Resource Allocation :266468 MB-seconds, 59 vcore-seconds
>         Diagnostics : N/A
>  
> [http://spark.rstudio.com/]
>  
> I can provide more details if required
>  
> Thanks,
> Bharath



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message