spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From yxzhao <yxz...@ualr.edu>
Subject Re: Spark Processing Large Data Stuck
Date Sat, 21 Jun 2014 22:27:23 GMT
Thanks Krishna,
I use a small cluster and each compute node has 16GB of RAM and 8 2.66GHz
CPU cores.









On Sat, Jun 21, 2014 at 3:16 PM, Krishna Sankar [via Apache Spark User
List] <ml-node+s1001560n8077h28@n3.nabble.com> wrote:

> Hi,
>
>    - I have seen similar behavior before. As far as I can tell, the root
>    cause is the out of memory error - verified this by monitoring the memory.
>       - I had a 30 GB file and was running on a single machine with 16GB.
>       So I knew it would fail.
>       - But instead of raising an exception, some part of the system
>       keeps on churning.
>    - My suggestion is to follow the memory settings for the JVM (try
>    bigger settings), make sure the settings are propagated to all the workers
>    and finally monitor the memory while the job is running.
>    - Another vector is to split the file, try with progressively
>    increasing size.
>    - I also see symptoms of failed connections. While I can't positively
>    say that it is a problem, check your topology & network connectivity.
>    - Out of curiosity, what kind of machines are you running ? Bare metal
>    ? EC2 ? How much memory ? 64 bit OS ?
>       - I assume these are big machines and so the resources themselves
>       might not be a problem.
>
> Cheers
> <k/>
>
>
> On Sat, Jun 21, 2014 at 12:55 PM, yxzhao <[hidden email]
> <http://user/SendEmail.jtp?type=node&node=8077&i=0>> wrote:
>
>> I run the pagerank example processing a large data set, 5GB in size,
>> using 48
>> machines. The job got stuck at the time point: 14/05/20 21:32:17, as the
>> attached log shows. It was stuck there for more than 10 hours and then I
>> killed it at last. But I did not find any information explaining why it
>> was
>> stuck. Any suggestions? Thanks.
>>
>> Spark_OK_48_pagerank.log
>> <
>> http://apache-spark-user-list.1001560.n3.nabble.com/file/n8075/Spark_OK_48_pagerank.log
>> >
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Processing-Large-Data-Stuck-tp8075.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>
>
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Processing-Large-Data-Stuck-tp8075p8077.html
>  To unsubscribe from Spark Processing Large Data Stuck, click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=8075&code=eXh6aGFvQHVhbHIuZWR1fDgwNzV8LTY0Mjc0NDkzMQ==>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Processing-Large-Data-Stuck-tp8075p8080.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Mime
View raw message