spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denny Lee <denny.g....@gmail.com>
Subject Re: Spark 1.3 SQL Programming Guide and sql._ / sql.types._
Date Fri, 20 Feb 2015 20:44:22 GMT
Oh no worries at all. If you want, I'd be glad to make updates and PR for
anything I find, eh?!
On Fri, Feb 20, 2015 at 12:18 Michael Armbrust <michael@databricks.com>
wrote:

> Yeah, sorry.  The programming guide has not been updated for 1.3.  I'm
> hoping to get to that this weekend / next week.
>
> On Fri, Feb 20, 2015 at 9:55 AM, Denny Lee <denny.g.lee@gmail.com> wrote:
>
>> Quickly reviewing the latest SQL Programming Guide
>> <https://github.com/apache/spark/blob/master/docs/sql-programming-guide.md>
>> (in github) I had a couple of quick questions:
>>
>> 1) Do we need to instantiate the SparkContext as per
>> // sc is an existing SparkContext.
>> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
>>
>> Within Spark 1.3 the sqlContext is already available so probably do not
>> need to make this call.
>>
>> 2) Importing org.apache.spark.sql._ should bring in both SQL data types,
>> struct types, and row
>> // Import Spark SQL data types and Row.
>> import org.apache.spark.sql._
>>
>> Currently with Spark 1.3 RC1, it appears org.apache.spark.sql._ only
>> brings in row.
>>
>> scala> import org.apache.spark.sql._
>>
>> import org.apache.spark.sql._
>>
>>
>> scala> val schema =
>>
>>      |   StructType(
>>
>>      |     schemaString.split(" ").map(fieldName =>
>> StructField(fieldName, StringType, true)))
>>
>> <console>:25: error: not found: value StructType
>>
>>          StructType(
>>
>> But if I also import in org.apache.spark.sql.types_
>>
>> scala> import org.apache.spark.sql.types._
>>
>> import org.apache.spark.sql.types._
>>
>>
>> scala> val schema =
>>
>>      |   StructType(
>>
>>      |     schemaString.split(" ").map(fieldName =>
>> StructField(fieldName, StringType, true)))
>>
>> schema: org.apache.spark.sql.types.StructType =
>> StructType(StructField(DeviceMake,StringType,true),
>> StructField(Country,StringType,true))
>>
>> Wondering if this is by design or perhaps a quick documentation / package
>> update is warranted.
>>
>>
>>
>>
>>
>

Mime
View raw message