spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pa Rö <paul.roewer1...@googlemail.com>
Subject Re: k-means iteration not terminate
Date Tue, 21 Jul 2015 08:42:49 GMT
thanks for this information, but i use cloudera live 5.4.4, and that have
only spark 1.3. a newer version is not avaible.
i don't understand this problem, first it compute some iterations and than
it "stop" better "do nothing". i think the problem
is not find in the program code.
maybe you know a other way to fix the problem?

2015-07-21 10:09 GMT+02:00 Akhil Das <akhil@sigmoidanalytics.com>:

> It could be a GC pause or something, you need to check in the stages tab
> and see what is taking time, If you upgrade to Spark 1.4, it has better UI
> and DAG visualization which helps you debug better.
>
> Thanks
> Best Regards
>
> On Mon, Jul 20, 2015 at 8:21 PM, Pa Rö <paul.roewer1990@googlemail.com>
> wrote:
>
>> hi community,
>>
>> i have write a spark k-means app. now i run it on a cluster.
>> my job start and at iteration nine or ten the process stop.
>> in the spark dashbord all time shown "is running", but nothing
>> happend, no exceptions.
>>
>> my setting is the following:
>> 1000 input points
>> k=10
>> maxIteration=30
>> a tree node cluster (one node have 16GB RAM und 8cores i7)
>> i use cloudera live 5.4.4 with spark 1.3
>>
>> maybe spark need more memory or i have a wrong setting?
>>
>> best regards,
>> paul
>>
>
>

Mime
View raw message