spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pa Rö <paul.roewer1...@googlemail.com>
Subject Re: spark vs flink low memory available
Date Tue, 11 Aug 2015 08:24:26 GMT
my first post is here and a log too:
http://mail-archives.us.apache.org/mod_mbox/spark-user/201508.mbox/%3CCAH2_pyKQhFR4tBvPbt2TDHGm+ZRkCBZfNK7ueDkJPdhe472fVQ@mail.gmail.com%3E

i use cloudera live, i think i can not use spark 1.5.

i will try to run it again and post the current logfile here.

2015-08-11 10:03 GMT+02:00 Ted Yu <yuzhihong@gmail.com>:

> Pa:
> Can you try 1.5.0 SNAPSHOT ?
>
> See SPARK-7075 Project Tungsten (Spark 1.5 Phase 1)
>
> Cheers
>
> On Tue, Aug 11, 2015 at 12:49 AM, jun <kitaev@126.com> wrote:
>
>> your detail of log file?
>>
>> At 2015-08-10 22:02:16, "Pa Rö" <paul.roewer1990@googlemail.com> wrote:
>>
>> hi community,
>>
>> i have build a spark and flink k-means application.
>> my test case is a clustering on 1 million points on 3node cluster.
>>
>> in memory bottlenecks begins flink to outsource to disk and work slowly
>> but works.
>> however spark lose executers if the memory is full and starts again
>> (infinety loop?).
>>
>> i try to customize the memory setting with the help from the mailing
>> list here
>> <http://mail-archives.us.apache.org/mod_mbox/spark-user/201508.mbox/%3CCAH2_pyKQhFR4tBvPbt2TDHGm+ZRkCBZfNK7ueDkJPdhe472fVQ@mail.gmail.com%3E>,
>> thanks.
>> but spark not work.
>>
>> is it necessary to have any configurations to be set? i mean flink work
>> with low memory, spark must also be able to or not?
>>
>> best regards,
>> paul
>>
>>
>>
>>
>

Mime
View raw message