spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pengcheng <pch...@gmail.com>
Subject Re: use CrossValidatorModel for prediction
Date Mon, 03 Oct 2016 02:12:53 GMT
After I used dataframe, sparkSQL, the dashboard showed bunch of SQLs.

Does that mean all the sql job are still running? The list is ever growing
and never stops.


thanks for any pointer!


peter



[image: Inline image 1]

On Sun, Oct 2, 2016 at 11:04 AM, Pengcheng Luo <pchluo@gmail.com> wrote:

>
>
> On Oct 2, 2016, at 1:04 AM, Pengcheng <pchluo@gmail.com> wrote:
>
> Dear Spark Users,
>
> I was wondering.
>
> I have a trained crossvalidator model
> *model: CrossValidatorModel*
>
> I wan to predict a score for  *features: RDD[Features]*
>
> Right now I have to convert features to dataframe and then perform
> predictions as following:
>
> """
>     val sqlContext = new SQLContext(features.context)
>     val input: DataFrame = sqlContext.createDataFrame(features.map(x =>
> (Vectors.dense(x.getArray),1.0) )).toDF("features", "label")
>     model.transform(input)
> """
>
> *i wonder If there is any API I can use to performance prediction on each
> individual Features*
>
> *For example, features.map( x => model.predict(x) ) *
>
>
>
> a big thank you!
>
> peter
>
>

Mime
View raw message