spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rajdeep Dua <rajdeep....@gmail.com>
Subject Re: InferredSchema Example in Spark-SQL
Date Mon, 18 May 2015 16:19:58 GMT
Thanks will try this.

On Sun, May 17, 2015 at 8:07 PM, Ram Sriharsha <sriharsha.ram@gmail.com>
wrote:

> you are missing sqlContext.implicits._
>
> On Sun, May 17, 2015 at 8:05 PM, Rajdeep Dua <rajdeep.dua@gmail.com>
> wrote:
>
>> Here are my imports
>>
>> *import* org.apache.spark.SparkContext
>>
>> *import* org.apache.spark.SparkContext._
>>
>> *import* org.apache.spark.SparkConf
>>
>> *import* org.apache.spark.sql.SQLContext
>>
>> *import* org.apache.spark.sql.SchemaRDD
>>
>> On Sun, May 17, 2015 at 8:05 PM, Rajdeep Dua <rajdeep.dua@gmail.com>
>> wrote:
>>
>>> Sorry .. toDF() gives an error
>>>
>>> [error]
>>> /home/ubuntu/work/spark/spark-samples/ml-samples/src/main/scala/sql/InferredSchema.scala:24:
>>> value toDF is not a member of org.apache.spark.rdd.RDD[Person]
>>> [error]     val people =
>>> sc.textFile("/home/ubuntu/work/spark-src/spark/examples/src/main/resources/people.txt").map(_.split(",")).map(p
>>> => Person(p(0), p(1).trim.toInt)).toDF()
>>>
>>>
>>> On Sun, May 17, 2015 at 5:41 PM, Cheng, Hao <hao.cheng@intel.com> wrote:
>>>
>>>>  Typo? Should be .toDF(), not .toRD()
>>>>
>>>>
>>>>
>>>> *From:* Ram Sriharsha [mailto:sriharsha.ram@gmail.com]
>>>> *Sent:* Monday, May 18, 2015 8:31 AM
>>>> *To:* Rajdeep Dua
>>>> *Cc:* user
>>>> *Subject:* Re: InferredSchema Example in Spark-SQL
>>>>
>>>>
>>>>
>>>> you mean toDF() ? (toDF converts the RDD to a DataFrame, in this case
>>>> inferring schema from the case class)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Sun, May 17, 2015 at 5:07 PM, Rajdeep Dua <rajdeep.dua@gmail.com>
>>>> wrote:
>>>>
>>>>  Hi All,
>>>>
>>>> Was trying the Inferred Schema spart example
>>>>
>>>> http://spark.apache.org/docs/latest/sql-programming-guide.html#overview
>>>>
>>>>
>>>>
>>>> I am getting the following compilation error on the function toRD()
>>>>
>>>>
>>>>
>>>> value toRD is not a member of org.apache.spark.rdd.RDD[Person]
>>>>
>>>> [error]     val people =
>>>> sc.textFile("/home/ubuntu/work/spark-src/spark/examples/src/main/resources/people.txt").map(_.split(",")).map(p
>>>> => Person(p(0), p(1).trim.toInt)).toRD()
>>>>
>>>> [error]
>>>>
>>>>
>>>>
>>>> Thanks
>>>>
>>>> Rajdeep
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>

Mime
View raw message