spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cheng, Hao" <hao.ch...@intel.com>
Subject RE: SparkSQL 1.1 hang when "DROP" or "LOAD"
Date Tue, 16 Sep 2014 04:25:47 GMT
The Hadoop client jar should be assembled into the uber-jar, but (I suspect) it's probably
not compatible with your Hadoop Cluster.
Can you also paste the Spark uber-jar name? Usually will be under the path lib/spark-assembly-1.1.0-xxx-hadoopxxx.jar.


-----Original Message-----
From: linkpatrickliu [mailto:linkpatrickliu@live.com] 
Sent: Tuesday, September 16, 2014 12:14 PM
To: user@spark.incubator.apache.org
Subject: RE: SparkSQL 1.1 hang when "DROP" or "LOAD"

Hi, Hao Cheng,

Here is the Spark\Hadoop version:
Spark version = 1.1.0
Hadoop version = 2.0.0-cdh4.6.0

And hive-site.xml:
<configuration>

  <property>
    <name>fs.default.name</name>
    <value>hdfs://ns</value>
  </property>
  <property>
    <name>dfs.nameservices</name>
    <value>ns</value>
  </property>
  
  <property>
    <name>dfs.ha.namenodes.ns</name>
    <value>machine01,machine02</value>
  </property>
  
  <property>
    <name>dfs.namenode.rpc-address.ns.machine01</name>
    <value>machine01:54310</value>
    
  </property>
  <property>
    <name>dfs.namenode.rpc-address.ns.machine02</name>
    <value>machine02:54310</value>
    
  </property>
  
  <property>
    <name>dfs.client.failover.proxy.provider.ns</name>
   
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://localhost:3306/metastore</value>
    <description>JDBC connect string for a JDBC metastore</description>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>Driver class name for a JDBC metastore</description>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hive_user</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>hive_123</value>
  </property>
  <property>
    <name>datanucleus.autoCreateSchema</name>
    <value>false</value>
  </property>
  <property>
    <name>datanucleus.autoCreateTables</name>
    <value>true</value>
  </property>
  <property>
    <name>datanucleus.fixedDatastore</name>
    <value>false</value>
  </property>
  <property>
    <name>hive.support.concurrency</name>
    <description>Enable Hive's Table Lock Manager Service</description>
    <value>true</value>
  </property>

  <property>
    <name>hive.zookeeper.quorum</name>
    <value>machine01,machine02,machine03</value>
    <description>Zookeeper quorum used by Hive's Table Lock Manager</description>
  </property>
  <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/user/hive/warehouse</value>
    <description>Hive warehouse directory</description>
  </property>
  <property>
    <name>mapred.job.tracker</name>
    <value>machine01:8032</value>
  </property>
  <property>
     <name>io.compression.codecs</name>
     <value>org.apache.hadoop.io.compress.SnappyCodec</value>
  </property>
  <property>
    <name>mapreduce.output.fileoutputformat.compress.codec</name>
    <value>org.apache.hadoop.io.compress.SnappyCodec</value>
  </property>
  <property>
    <name>mapreduce.output.fileoutputformat.compress.type</name>
    <value>BLOCK</value>
  </property>
  <property>
    <name>hive.exec.show.job.failure.debug.info</name>
    <value>true</value>
    <description>
    If a job fails, whether to provide a link in the CLI to the task with the
    most failures, along with debugging hints if applicable.
    </description>
  </property>
  <property>
    <name>hive.hwi.listen.host</name>
    <value>0.0.0.0</value>
    <description>This is the host address the Hive Web Interface will listen on</description>
  </property>
  <property>
    <name>hive.hwi.listen.port</name>
    <value>9999</value>
    <description>This is the port the Hive Web Interface will listen on</description>
  </property>
  <property>
    <name>hive.hwi.war.file</name>
    <value>/lib/hive-hwi-0.10.0-cdh4.2.0.war</value>
    <description>This is the WAR file with the jsp content for Hive Web Interface</description>
  </property>
  <property>
    <name>hive.aux.jars.path</name>
   
<value>file:///usr/lib/hive/lib/hive-hbase-handler-0.10.0-cdh4.6.0.jar,file:///usr/lib/hbase/hbase-0.94.15-cdh4.6.0-security.jar,file:///usr/lib/zookeeper/zookeeper.jar</value>
  </property>
  <property>
     <name>hbase.zookeeper.quorum</name>
     <value>machine01,machine02,machine03</value>
  </property>
  <property>
    <name>hive.cli.print.header</name>
    <value>true</value>
  </property>
  <property>
    <name>hive.metastore.execute.setugi</name>
    <value>true</value>
    <description>In unsecure mode, setting this property to true will cause the metastore
to execute DFS operations using the client's reported user and group permissions. Note that
this property must be set on both the client and server sides. Further note that its best
effort. If client sets its to true and server sets it to false, client setting will be ignored.</description>
  </property>
  <property>
    <name>hive.security.authorization.enabled</name>
    <value>true</value>
    <description>enable or disable the hive client authorization</description>
  </property>
  <property>
    <name>hive.metastore.authorization.storage.checks</name>
    <value>true</value>
  </property>
  <property>
    <name>hive.security.authorization.createtable.owner.grants</name>
    <value>ALL</value>
    <description>the privileges automatically granted to the owner whenever a table
gets created.
    An example like "select,drop" will grant select and drop privilege to the owner of the
table</description>
  </property>
</configuration>



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkSQL-1-1-hang-when-DROP-or-LOAD-tp14222p14320.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Mime
View raw message