spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yusuf Can Gürkan <yu...@useinsider.com>
Subject Re: SQLContext Create Table Problem
Date Wed, 19 Aug 2015 16:03:07 GMT
Hey Yin,

Thanks for answer. I thought that this could be problem but i can not create HiveContext because
i can not import org.apache.spark.sql.hive.HiveContext. It does not see this package. 

I read that i should build spark with -PHive but i’m running on Amazon EMR 1.4.1 and on
spark-shell i can import hive package but can not do the same on spark-submit. Do you have
any idea why? Because if it’s related to build with -PHive, how can i import it in spark-shell?

> On 19 Aug 2015, at 18:59, Yin Huai <yhuai@databricks.com> wrote:
> 
> Can you try to use HiveContext instead of SQLContext? Your query is trying to create
a table and persist the metadata of the table in metastore, which is only supported by HiveContext.
> 
> On Wed, Aug 19, 2015 at 8:44 AM, Yusuf Can Gürkan <yusuf@useinsider.com <mailto:yusuf@useinsider.com>>
wrote:
> Hello,
> 
> I’m trying to create a table with sqlContext.sql method as below:
> 
> val sc = new SparkContext()
> val sqlContext = new SQLContext(sc)
> 
> import sqlContext.implicits._
> 
> sqlContext.sql(s"""
> create table if not exists landing (
> date string,
> referrer string
> )
> partitioned by (partnerid string,dt string)
> row format delimited fields terminated by '\t' lines terminated by '\n'
> STORED AS TEXTFILE LOCATION 's3n://...' <>
>       "”")
> 
> 
> It gives error on spark-submit:
> 
> Exception in thread "main" java.lang.RuntimeException: [2.1] failure: ``with'' expected
but identifier create found
> 
> create external table if not exists landing (
> 
> ^
> 	at scala.sys.package$.error(package.scala:27)
> 	at org.apache.spark.sql.catalyst.AbstractSparkSQLParser.parse(AbstractSparkSQLParser.scala:36)
> 	at org.apache.spark.sql.catalyst.DefaultParserDialect.parse(ParserDialect.scala:67)
> 
> 
> 
> What can be the reason??
> 


Mime
View raw message