spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From masaki rikitoku <>
Subject number of partitions for hive schemaRDD
Date Thu, 26 Feb 2015 09:31:44 GMT
Hi all

now, I'm trying the SparkSQL with hivecontext.

when I execute the hql like the following.


val ctx = new org.apache.spark.sql.hive.HiveContext(sc)
import ctx._

val queries = ctx.hql("select keyword from queries where dt =
'2015-02-01' limit 10000000")


It seem that the number of the partitions ot the queries is set by 1.

Is this the specifications for schemaRDD, SparkSQL, HiveContext ?

Are there any means to set the number of partitions arbitrary value
except for explicit repartition

Masaki Rikitoku

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message