Checked default is gzip



Dr Mich Talebzadeh

 

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 


On 3 July 2016 at 23:39, Mich Talebzadeh <mich.talebzadeh@gmail.com> wrote:
thanks Ted that was it :)

scala> val c = sqlContext.setConf("spark.sql.parquet.compression.codec", "uncompressed")
c: Unit = ()
scala> val s4 = s.write.mode("overwrite").parquet("/user/hduser/sales4")
s4: Unit = ()


Before
-rw-r--r--   2 hduser supergroup      17487 2016-07-03 22:28 /user/hduser/sales4/part-r-00199-9dcd4fb8-148d-48ba-9da3-8d68aa24aa5c.gz.parquet

After

hduser@rhes564:: :/home/hduser/dba/bin/sales> hdfs dfs -ls /user/hduser/sales4
-rw-r--r--   2 hduser supergroup      40190 2016-07-03 23:23 /user/hduser/sales4/part-r-00000-19100306-f3d6-44fb-8bde-55307101cf3f.parquet


Now the question is that if you do not specify the compression  with setConf it default to gzip compression?

val s4 = s.write.mode("overwrite").parquet("/user/hduser/sales4")


Cheers

Dr Mich Talebzadeh

 

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.

 


On 3 July 2016 at 23:21, Ted Yu <yuzhihong@gmail.com> wrote:
Have you tried the following (note the extraneous dot in your config name) ?

val c = sqlContext.setConf("spark.sql.parquet.compression.codec", "none")

Also, parquet() has compression parameter which defaults to None

FYI

On Sun, Jul 3, 2016 at 2:42 PM, Mich Talebzadeh <mich.talebzadeh@gmail.com> wrote:
Hi,

I simply read a Parquet table

scala> val s = sqlContext.read.parquet("oraclehadoop.sales2")
s: org.apache.spark.sql.DataFrame = [prod_id: bigint, cust_id: bigint, time_id: timestamp, channel_id: bigint, promo_id: bigint, quantity_sold: decimal(10,0), amount_sold: decimal(10,0)]

Now all I want is to save data and make it uncompressed. By default it saves the table as gzipped

val s4 = s.write.mode("overwrite").parquet("/user/hduser/sales4")

However, I want use this approach without creating table explicitly myself with sqlContext etc

This does not seem to work

val c = sqlContext.setConf("spark.sql.parquet.compression.codec.", "uncompressed")

Can I do through a method on DataFrame "s" above to make the table saved as uncompressed?

Thanks,

Dr Mich Talebzadeh

 

LinkedIn  https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com


Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.