spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shivaram Venkataraman <shiva...@eecs.berkeley.edu>
Subject Re: Using TF-IDF from MLlib
Date Tue, 17 Mar 2015 01:45:10 GMT
FWIW the JIRA I was thinking about is
https://issues.apache.org/jira/browse/SPARK-3098

On Mon, Mar 16, 2015 at 6:10 PM, Shivaram Venkataraman <
shivaram@eecs.berkeley.edu> wrote:

> I vaguely remember that JIRA and AFAIK Matei's point was that the order is
> not guaranteed *after* a shuffle. If you only use operations like map which
> preserve partitioning, ordering should be guaranteed from what I know.
>
> On Mon, Mar 16, 2015 at 6:06 PM, Sean Owen <sowen@cloudera.com> wrote:
>
>> Dang I can't seem to find the JIRA now but I am sure we had a discussion
>> with Matei about this and the conclusion was that RDD order is not
>> guaranteed unless a sort is involved.
>> On Mar 17, 2015 12:14 AM, "Joseph Bradley" <joseph@databricks.com> wrote:
>>
>>> This was brought up again in
>>> https://issues.apache.org/jira/browse/SPARK-6340  so I'll answer one
>>> item which was asked about the reliability of zipping RDDs.  Basically, it
>>> should be reliable, and if it is not, then it should be reported as a bug.
>>> This general approach should work (with explicit types to make it clear):
>>>
>>> val data: RDD[LabeledPoint] = ...
>>> val labels: RDD[Double] = data.map(_.label)
>>> val features1: RDD[Vector] = data.map(_.features)
>>> val features2: RDD[Vector] = new
>>> HashingTF(numFeatures=100).transform(features1)
>>> val features3: RDD[Vector] = idfModel.transform(features2)
>>> val finalData: RDD[LabeledPoint] = labels.zip(features3).map((label,
>>> features) => LabeledPoint(label, features))
>>>
>>> If you run into problems with zipping like this, please report them!
>>>
>>> Thanks,
>>> Joseph
>>>
>>> On Mon, Dec 29, 2014 at 4:06 PM, Xiangrui Meng <mengxr@gmail.com> wrote:
>>>
>>>> Hopefully the new pipeline API addresses this problem. We have a code
>>>> example here:
>>>>
>>>>
>>>> https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/ml/SimpleTextClassificationPipeline.scala
>>>>
>>>> -Xiangrui
>>>>
>>>> On Mon, Dec 29, 2014 at 5:22 AM, andy petrella <andy.petrella@gmail.com>
>>>> wrote:
>>>> > Here is what I did for this case :
>>>> https://github.com/andypetrella/tf-idf
>>>> >
>>>> >
>>>> > Le lun 29 déc. 2014 11:31, Sean Owen <sowen@cloudera.com> a écrit
:
>>>> >
>>>> >> Given (label, terms) you can just transform the values to a TF
>>>> vector,
>>>> >> then TF-IDF vector, with HashingTF and IDF / IDFModel. Then you
can
>>>> >> make a LabeledPoint from (label, vector) pairs. Is that what you're
>>>> >> looking for?
>>>> >>
>>>> >> On Mon, Dec 29, 2014 at 3:37 AM, Yao <yge@ford.com> wrote:
>>>> >> > I found the TF-IDF feature extraction and all the MLlib code
that
>>>> work
>>>> >> > with
>>>> >> > pure Vector RDD very difficult to work with due to the lack
of
>>>> ability
>>>> >> > to
>>>> >> > associate vector back to the original data. Why can't Spark
MLlib
>>>> >> > support
>>>> >> > LabeledPoint?
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > --
>>>> >> > View this message in context:
>>>> >> >
>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Using-TF-IDF-from-MLlib-tp19429p20876.html
>>>> >> > Sent from the Apache Spark User List mailing list archive at
>>>> Nabble.com.
>>>> >> >
>>>> >> >
>>>> ---------------------------------------------------------------------
>>>> >> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> >> > For additional commands, e-mail: user-help@spark.apache.org
>>>> >> >
>>>> >>
>>>> >> ---------------------------------------------------------------------
>>>> >> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> >> For additional commands, e-mail: user-help@spark.apache.org
>>>> >>
>>>> >
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>
>>>>
>>>
>

Mime
View raw message