spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yusuf Can Gürkan <yu...@useinsider.com>
Subject Re: SQLContext Create Table Problem
Date Thu, 20 Aug 2015 09:22:25 GMT
Someone has any idea about this subject?

> On 20 Aug 2015, at 00:47, Yusuf Can Gürkan <yusuf@useinsider.com> wrote:
> 
> Hey,
> 
> This is my spark-env:
> 
> # Add Hadoop libraries to Spark classpath
> SPARK_CLASSPATH="${SPARK_CLASSPATH}:${HADOOP_HOME}/*:${HADOOP_HOME}/../hadoop-hdfs/*:${HADOOP_HOME}/../hadoop-mapreduce/*:${HADOOP_HOME}/../hadoop-yarn/*"
> LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${HADOOP_HOME}/lib/native"
> 
> # Add Hive conf to Spark classpath
> SPARK_CLASSPATH="${SPARK_CLASSPATH}:${HIVE_CONF_DIR}"
> 
> # Add Hadoop LZO library to Spark classpath and native library to library path
> SPARK_CLASSPATH="${SPARK_CLASSPATH}:${HADOOP_HOME}/../hadoop-lzo/lib/*"
> LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${HADOOP_HOME}/../hadoop-lzo/lib/native"
> 
> # Add EMRFS libraries to Spark classpath
> SPARK_CLASSPATH="${SPARK_CLASSPATH}:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*"
> 
> export SPARK_CLASSPATH
> export LD_LIBRARY_PATH
> 
> I think hive is on class path, should i do another thing?
> 
>> On 19 Aug 2015, at 20:12, Eric Walker <eric@node.io <mailto:eric@node.io>>
wrote:
>> 
>> Hi,
>> 
>> On EMR, take a look at the "SPARK_CLASSPATH" variable in /home/hadoop/spark/conf/spark-env.sh.
 You may need to add a jar file to it.
>> 
>> Regards,
>> Eric
>> 
>> 
>> On Wed, Aug 19, 2015 at 11:03 AM, Yusuf Can Gürkan <yusuf@useinsider.com <mailto:yusuf@useinsider.com>>
wrote:
>> Hey Yin,
>> 
>> Thanks for answer. I thought that this could be problem but i can not create HiveContext
because i can not import org.apache.spark.sql.hive.HiveContext. It does not see this package.

>> 
>> I read that i should build spark with -PHive but i’m running on Amazon EMR 1.4.1
and on spark-shell i can import hive package but can not do the same on spark-submit. Do you
have any idea why? Because if it’s related to build with -PHive, how can i import it in
spark-shell?
>> 
>>> On 19 Aug 2015, at 18:59, Yin Huai <yhuai@databricks.com <mailto:yhuai@databricks.com>>
wrote:
>>> 
>>> Can you try to use HiveContext instead of SQLContext? Your query is trying to
create a table and persist the metadata of the table in metastore, which is only supported
by HiveContext.
>>> 
>>> On Wed, Aug 19, 2015 at 8:44 AM, Yusuf Can Gürkan <yusuf@useinsider.com <mailto:yusuf@useinsider.com>>
wrote:
>>> Hello,
>>> 
>>> I’m trying to create a table with sqlContext.sql method as below:
>>> 
>>> val sc = new SparkContext()
>>> val sqlContext = new SQLContext(sc)
>>> 
>>> import sqlContext.implicits._
>>> 
>>> sqlContext.sql(s"""
>>> create table if not exists landing (
>>> date string,
>>> referrer string
>>> )
>>> partitioned by (partnerid string,dt string)
>>> row format delimited fields terminated by '\t' lines terminated by '\n'
>>> STORED AS TEXTFILE LOCATION 's3n://...' <>
>>>       "”")
>>> 
>>> 
>>> It gives error on spark-submit:
>>> 
>>> Exception in thread "main" java.lang.RuntimeException: [2.1] failure: ``with''
expected but identifier create found
>>> 
>>> create external table if not exists landing (
>>> 
>>> ^
>>> 	at scala.sys.package$.error(package.scala:27)
>>> 	at org.apache.spark.sql.catalyst.AbstractSparkSQLParser.parse(AbstractSparkSQLParser.scala:36)
>>> 	at org.apache.spark.sql.catalyst.DefaultParserDialect.parse(ParserDialect.scala:67)
>>> 
>>> 
>>> 
>>> What can be the reason??
>>> 
>> 
>> 
> 


Mime
View raw message