spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Armbrust <mich...@databricks.com>
Subject Re: Spark 1.3 SQL Programming Guide and sql._ / sql.types._
Date Fri, 20 Feb 2015 20:18:36 GMT
Yeah, sorry.  The programming guide has not been updated for 1.3.  I'm
hoping to get to that this weekend / next week.

On Fri, Feb 20, 2015 at 9:55 AM, Denny Lee <denny.g.lee@gmail.com> wrote:

> Quickly reviewing the latest SQL Programming Guide
> <https://github.com/apache/spark/blob/master/docs/sql-programming-guide.md>
> (in github) I had a couple of quick questions:
>
> 1) Do we need to instantiate the SparkContext as per
> // sc is an existing SparkContext.
> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
>
> Within Spark 1.3 the sqlContext is already available so probably do not
> need to make this call.
>
> 2) Importing org.apache.spark.sql._ should bring in both SQL data types,
> struct types, and row
> // Import Spark SQL data types and Row.
> import org.apache.spark.sql._
>
> Currently with Spark 1.3 RC1, it appears org.apache.spark.sql._ only
> brings in row.
>
> scala> import org.apache.spark.sql._
>
> import org.apache.spark.sql._
>
>
> scala> val schema =
>
>      |   StructType(
>
>      |     schemaString.split(" ").map(fieldName => StructField(fieldName,
> StringType, true)))
>
> <console>:25: error: not found: value StructType
>
>          StructType(
>
> But if I also import in org.apache.spark.sql.types_
>
> scala> import org.apache.spark.sql.types._
>
> import org.apache.spark.sql.types._
>
>
> scala> val schema =
>
>      |   StructType(
>
>      |     schemaString.split(" ").map(fieldName => StructField(fieldName,
> StringType, true)))
>
> schema: org.apache.spark.sql.types.StructType =
> StructType(StructField(DeviceMake,StringType,true),
> StructField(Country,StringType,true))
>
> Wondering if this is by design or perhaps a quick documentation / package
> update is warranted.
>
>
>
>
>

Mime
View raw message