spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mich Talebzadeh <>
Subject Re: How spark makes partition when we insert data using the Sql query, and how the permissions to the partitions is assigned.?
Date Fri, 01 Jul 2016 08:40:40 GMT
Let us take this for a ride.

Simple code. Reads from an existing of 22miilion rows stored as ORC and
saves it as a Parquet

val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
HiveContext.sql("use oraclehadoop")
val s = HiveContext.table("sales2")
val sorted = s.sort("prod_id","cust_id","time_id","channel_id","promo_id")

It will store it on hdfs in this case, under directory

Note the subdirectory corresponds to save()

It is saved by Spark and the number of partitions I gather is determined by
Spark code (I don't know the content)

The sub-directory owner on hdfs will be /user/<LINUX_USER_NAME> by default
and if you list the partitions it will show something like below:

-rw-r--r--   2 hduser supergroup          0 2016-07-01 09:26
-rw-r--r--   2 hduser supergroup        743 2016-07-01 09:26
-rw-r--r--   2 hduser supergroup     182639 2016-07-01 09:26
-rw-r--r--   2 hduser supergroup      22962 2016-07-01 09:23
-rw-r--r--   2 hduser supergroup      25698 2016-07-01 09:23
-rw-r--r--   2 hduser supergroup      17210 2016-07-01 09:23
-rw-r--r--   2 hduser supergroup      22398 2016-07-01 09:23
-rw-r--r--   2 hduser supergroup      18105 2016-07-01 09:23

Note that metadata is also stored on the file together with data partitions
(zipped). This is in contrast to Hive where metadata is stored in Hive

In this case it decides to have 200 data partitions.

If you want more control of it, you can get the data as DF, register it as
tempTable, create the table the way you like it and do an insert/select
from tempTable.

I personally prefer to create the table myself in a format that I like
(liker ORC below) and store it in Hive.


var sqltext: String = ""
sqltext =
 CREATE TABLE IF NOT EXISTS oraclehadoop.sales3
  PROD_ID        bigint                       ,
  CUST_ID        bigint                       ,
  TIME_ID        timestamp                    ,
  CHANNEL_ID     bigint                       ,
  PROMO_ID       bigint                       ,
  QUANTITY_SOLD  decimal(10)                  ,
  AMOUNT_SOLD    decimal(10)
TBLPROPERTIES ( "orc.compress"="SNAPPY",
sqltext =


Dr Mich Talebzadeh

LinkedIn *

*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.

On 1 July 2016 at 08:43, shiv4nsh <> wrote:

> Hey guys I am using Apache Spark 1.5.2, and I am running the Sql query
> using
> the SQLContext and when I run the insert query it saves the data in
> partition (as expected).
> I am just curious and want to know how these partitions are made and how
> the
> permissions to these partition is assigned . Can we change it? Does it
> behave differently on hdfs.?
> If someone can point me to the exact code in spark  that would be
> beneficial.
> I have also posted it on  stackOverflow.
> <
> >
> --
> View this message in context:
> Sent from the Apache Spark User List mailing list archive at
> ---------------------------------------------------------------------
> To unsubscribe e-mail:

View raw message