spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: error creating custom schema
Date Wed, 23 Dec 2015 10:43:43 GMT
Looks like a comma was missing after "C1"

Cheers

> On Dec 23, 2015, at 1:47 AM, Divya Gehlot <divya.htconex@gmail.com> wrote:
> 
> Hi,
> I am trying to create custom schema but its throwing below error 
> 
> 
>> scala> import org.apache.spark.sql.hive.HiveContext
>> import org.apache.spark.sql.hive.HiveContext
>> 
>> scala> import org.apache.spark.sql.hive.orc._
>> import org.apache.spark.sql.hive.orc._
>> 
>> scala> val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
>> 15/12/23 04:42:09 WARN SparkConf: The configuration key 'spark.yarn.applicationMaster.waitTries'
has been deprecated as of Spark 1.3 and and may be removed in the future. Please use the new
key 'spark.yarn.am.waitTime' instead.
>> 15/12/23 04:42:09 INFO HiveContext: Initializing execution hive, version 0.13.1
>> hiveContext: org.apache.spark.sql.hive.HiveContext = org.apache.spark.sql.hive.HiveContext@3ca50ddf
>> 
>> scala> import org.apache.spark.sql.types.{StructType, StructField, StringType,
IntegerType,FloatType ,LongType ,TimestampType };
>> import org.apache.spark.sql.types.{StructType, StructField, StringType, IntegerType,
FloatType, LongType, TimestampType}
>> 
>> scala> val loandepoSchema = StructType(Seq(
>>      | StructField("C1" StringType, true),
>>      | StructField("COLUMN2", StringType , true),
>>      | StructField("COLUMN3", StringType, true),
>>      | StructField("COLUMN4", StringType, true),
>>      | StructField("COLUMN5", StringType , true),
>>      | StructField("COLUMN6", StringType, true),
>>      | StructField("COLUMN7", StringType, true),
>>      | StructField("COLUMN8", StringType, true),
>>      | StructField("COLUMN9", StringType, true),
>>      | StructField("COLUMN10", StringType, true),
>>      | StructField("COLUMN11", StringType, true),
>>      | StructField("COLUMN12", StringType, true),
>>      | StructField("COLUMN13", StringType, true),
>>      | StructField("COLUMN14", StringType, true),
>>      | StructField("COLUMN15", StringType, true),
>>      | StructField("COLUMN16", StringType, true),
>>      | StructField("COLUMN17", StringType, true)
>>      | StructField("COLUMN18", StringType, true),
>>      | StructField("COLUMN19", StringType, true),
>>      | StructField("COLUMN20", StringType, true),
>>      | StructField("COLUMN21", StringType, true),
>>      | StructField("COLUMN22", StringType, true)))
>> <console>:25: error: value StringType is not a member of String
>>        StructField("C1" StringType, true),
>>                         ^
> 
> Would really appreciate the guidance/pointers.
> 
> Thanks,
> Divya 

Mime
View raw message