spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay <jayadeep.jayara...@gmail.com>
Subject Re: [PySpark] Releasing memory after a spark job is finished
Date Tue, 05 Jun 2018 02:41:56 GMT
Can you tell us what version of Spark you are using and if Dynamic
Allocation is enabled ?

Also, how are the files being read ? Is it a single read of all files using
a file matching regex or are you running different threads in the same
pyspark job?



On Mon 4 Jun, 2018, 1:27 PM Shuporno Choudhury, <
shuporno.choudhury@gmail.com> wrote:

> Thanks a lot for the insight.
> Actually I have the exact same transformations for all the datasets, hence
> only 1 python code.
> Now, do you suggest that I run different spark-submit for all the
> different datasets given that I have the exact same transformations?
>
> On Tue 5 Jun, 2018, 1:48 AM Jörn Franke [via Apache Spark User List], <
> ml+s1001560n32458h84@n3.nabble.com> wrote:
>
>> Yes if they are independent with different transformations then I would
>> create a separate python program. Especially for big data processing
>> frameworks one should avoid to put everything in one big monotholic
>> applications.
>>
>>
>> On 4. Jun 2018, at 22:02, Shuporno Choudhury <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=32458&i=0>> wrote:
>>
>> Hi,
>>
>> Thanks for the input.
>> I was trying to get the functionality first, hence I was using local
>> mode. I will be running on a cluster definitely but later.
>>
>> Sorry for my naivety, but can you please elaborate on the modularity
>> concept that you mentioned and how it will affect whatever I am already
>> doing?
>> Do you mean running a different spark-submit for each different dataset
>> when you say 'an independent python program for each process '?
>>
>> On Tue, 5 Jun 2018 at 01:12, Jörn Franke [via Apache Spark User List] <[hidden
>> email] <http:///user/SendEmail.jtp?type=node&node=32458&i=1>> wrote:
>>
>> Why don’t you modularize your code and write for each process an
>>> independent python program that is submitted via Spark?
>>>
>>> Not sure though if Spark local make sense. If you don’t have a cluster
>>> then a normal python program can be much better.
>>>
>>> On 4. Jun 2018, at 21:37, Shuporno Choudhury <[hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=32455&i=0>> wrote:
>>>
>>> Hi everyone,
>>> I am trying to run a pyspark code on some data sets sequentially [basically
>>> 1. Read data into a dataframe 2.Perform some join/filter/aggregation 3.
>>> Write modified data in parquet format to a target location]
>>> Now, while running this pyspark code across *multiple independent data
>>> sets sequentially*, the memory usage from the previous data set doesn't
>>> seem to get released/cleared and hence spark's memory consumption (JVM
>>> memory consumption from Task Manager) keeps on increasing till it fails at
>>> some data set.
>>> So, is there a way to clear/remove dataframes that I know are not going
>>> to be used later?
>>> Basically, can I clear out some memory programmatically (in the pyspark
>>> code) when processing for a particular data set ends?
>>> At no point, I am caching any dataframe (so unpersist() is also not a
>>> solution).
>>>
>>> I am running spark using local[*] as master. There is a single
>>> SparkSession that is doing all the processing.
>>> If it is not possible to clear out memory, what can be a better approach
>>> for this problem?
>>>
>>> Can someone please help me with this and tell me if I am going wrong
>>> anywhere?
>>>
>>> --Thanks,
>>> Shuporno Choudhury
>>>
>>>
>>>
>>> ------------------------------
>>> If you reply to this email, your message will be added to the discussion
>>> below:
>>>
>>> http://apache-spark-user-list.1001560.n3.nabble.com/PySpark-Releasing-memory-after-a-spark-job-is-finished-tp32454p32455.html
>>>
>> To start a new topic under Apache Spark User List, email [hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=32458&i=2>
>>>
>> To unsubscribe from Apache Spark User List, click here.
>>> NAML
>>> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>
>>
>> --
>> --Thanks,
>> Shuporno Choudhury
>>
>>
>>
>> ------------------------------
>>
> If you reply to this email, your message will be added to the discussion
>> below:
>>
>
>> http://apache-spark-user-list.1001560.n3.nabble.com/PySpark-Releasing-memory-after-a-spark-job-is-finished-tp32454p32458.html
>>
> To start a new topic under Apache Spark User List, email
>> ml+s1001560n1h17@n3.nabble.com
>> To unsubscribe from Apache Spark User List, click here
>> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=c2h1cG9ybm8uY2hvdWRodXJ5QGdtYWlsLmNvbXwxfC0xODI0MTU0MzQ0>
>> .
>> NAML
>> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>

Mime
View raw message