spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nirmal Fernando <nir...@wso2.com>
Subject Re: [MLLib][Kmeans] KMeansModel.computeCost takes lot of time
Date Tue, 14 Jul 2015 03:30:56 GMT
Thanks Burak.

Now it takes minutes to repartition;

Active Stages (1) Stage IdDescriptionSubmittedDurationTasks: Succeeded/Total
InputOutputShuffle Read Shuffle Write  42 (kill)
<http://localhost:4040/stages/stage/kill/?id=42&terminate=true> repartition
at UnsupervisedSparkModelBuilder.java:120
<http://localhost:4040/stages/stage?id=42&attempt=0> +details

org.apache.spark.api.java.JavaRDD.repartition(JavaRDD.scala:100)
org.wso2.carbon.ml.core.spark.algorithms.UnsupervisedSparkModelBuilder.buildKMeansModel(UnsupervisedSparkModelBuilder.java:120)
org.wso2.carbon.ml.core.spark.algorithms.UnsupervisedSparkModelBuilder.build(UnsupervisedSparkModelBuilder.java:84)
org.wso2.carbon.ml.core.impl.MLModelHandler$ModelBuilder.run(MLModelHandler.java:576)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)

 2015/07/14 08:59:30 3.6 min
 0/3
 14.6 MB Pending Stages (1) Stage IdDescriptionSubmittedDurationTasks:
Succeeded/TotalInputOutputShuffle Read Shuffle Write  43 sum at
KMeansModel.scala:70
<http://localhost:4040/stages/stage?id=43&attempt=0> +details


org.apache.spark.rdd.DoubleRDDFunctions.sum(DoubleRDDFunctions.scala:33)
org.apache.spark.mllib.clustering.KMeansModel.computeCost(KMeansModel.scala:70)
org.wso2.carbon.ml.core.spark.algorithms.UnsupervisedSparkModelBuilder.buildKMeansModel(UnsupervisedSparkModelBuilder.java:121)
org.wso2.carbon.ml.core.spark.algorithms.UnsupervisedSparkModelBuilder.build(UnsupervisedSparkModelBuilder.java:84)
org.wso2.carbon.ml.core.impl.MLModelHandler$ModelBuilder.run(MLModelHandler.java:576)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)

 Unknown Unknown
 0/8

On Mon, Jul 13, 2015 at 11:44 PM, Burak Yavuz <brkyvz@gmail.com> wrote:

> Can you call repartition(8) or 16 on data.rdd(), before KMeans, and also,
> .cache()?
>
> something like, (I'm assuming you are using Java):
> ```
> JavaRDD<Vector> input = data.repartition(8).cache();
> org.apache.spark.mllib.clustering.KMeans.train(input.rdd(), 3, 20);
> ```
>
> On Mon, Jul 13, 2015 at 11:10 AM, Nirmal Fernando <nirmal@wso2.com> wrote:
>
>> I'm using;
>>
>> org.apache.spark.mllib.clustering.KMeans.train(data.rdd(), 3, 20);
>>
>> Cpu cores: 8 (using default Spark conf thought)
>>
>> On partitions, I'm not sure how to find that.
>>
>> On Mon, Jul 13, 2015 at 11:30 PM, Burak Yavuz <brkyvz@gmail.com> wrote:
>>
>>> What are the other parameters? Are you just setting k=3? What about # of
>>> runs? How many partitions do you have? How many cores does your machine
>>> have?
>>>
>>> Thanks,
>>> Burak
>>>
>>> On Mon, Jul 13, 2015 at 10:57 AM, Nirmal Fernando <nirmal@wso2.com>
>>> wrote:
>>>
>>>> Hi Burak,
>>>>
>>>> k = 3
>>>> dimension = 785 features
>>>> Spark 1.4
>>>>
>>>> On Mon, Jul 13, 2015 at 10:28 PM, Burak Yavuz <brkyvz@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> How are you running K-Means? What is your k? What is the dimension of
>>>>> your dataset (columns)? Which Spark version are you using?
>>>>>
>>>>> Thanks,
>>>>> Burak
>>>>>
>>>>> On Mon, Jul 13, 2015 at 2:53 AM, Nirmal Fernando <nirmal@wso2.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> For a fairly large dataset, 30MB, KMeansModel.computeCost takes lot
>>>>>> of time (16+ mints).
>>>>>>
>>>>>> It takes lot of time at this task;
>>>>>>
>>>>>> org.apache.spark.rdd.DoubleRDDFunctions.sum(DoubleRDDFunctions.scala:33)
>>>>>> org.apache.spark.mllib.clustering.KMeansModel.computeCost(KMeansModel.scala:70)
>>>>>>
>>>>>> Can this be improved?
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Thanks & regards,
>>>>>> Nirmal
>>>>>>
>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc.
>>>>>> Mobile: +94715779733
>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Thanks & regards,
>>>> Nirmal
>>>>
>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc.
>>>> Mobile: +94715779733
>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>>
>> Thanks & regards,
>> Nirmal
>>
>> Associate Technical Lead - Data Technologies Team, WSO2 Inc.
>> Mobile: +94715779733
>> Blog: http://nirmalfdo.blogspot.com/
>>
>>
>>
>


-- 

Thanks & regards,
Nirmal

Associate Technical Lead - Data Technologies Team, WSO2 Inc.
Mobile: +94715779733
Blog: http://nirmalfdo.blogspot.com/

Mime
View raw message