spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mich Talebzadeh <mich.talebza...@gmail.com>
Subject Re: Saving parquet table as uncompressed with write.mode("overwrite").
Date Sun, 03 Jul 2016 22:39:43 GMT
thanks Ted that was it :)

scala> val c = sqlContext.setConf("spark.sql.parquet.compression.codec",
"uncompressed")
c: Unit = ()
scala> val s4 = s.write.mode("overwrite").parquet("/user/hduser/sales4")
s4: Unit = ()


Before
-rw-r--r--   2 hduser supergroup      17487 2016-07-03 22:28
/user/hduser/sales4/part-r-00199-9dcd4fb8-148d-48ba-9da3-8d68aa24aa5c.*gz.*
parquet

After

hduser@rhes564:: :/home/hduser/dba/bin/sales> hdfs dfs -ls
/user/hduser/sales4
-rw-r--r--   2 hduser supergroup      40190 2016-07-03 23:23
/user/hduser/sales4/part-r-00000-19100306-f3d6-44fb-8bde-55307101cf3f.parquet


Now the question is that if you do not specify the compression  with
setConf it default to gzip compression?

val s4 = s.write.mode("overwrite").parquet("/user/hduser/sales4")


Cheers

Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 3 July 2016 at 23:21, Ted Yu <yuzhihong@gmail.com> wrote:

> Have you tried the following (note the extraneous dot in your config name)
> ?
>
> val c = sqlContext.setConf("spark.sql.parquet.compression.codec", "none")
>
> Also, parquet() has compression parameter which defaults to None
>
> FYI
>
> On Sun, Jul 3, 2016 at 2:42 PM, Mich Talebzadeh <mich.talebzadeh@gmail.com
> > wrote:
>
>> Hi,
>>
>> I simply read a Parquet table
>>
>> scala> val s = sqlContext.read.parquet("oraclehadoop.sales2")
>> s: org.apache.spark.sql.DataFrame = [prod_id: bigint, cust_id: bigint,
>> time_id: timestamp, channel_id: bigint, promo_id: bigint, quantity_sold:
>> decimal(10,0), amount_sold: decimal(10,0)]
>>
>> Now all I want is to save data and make it uncompressed. By default it
>> saves the table as *gzipped*
>>
>> val s4 = s.write.mode("overwrite").parquet("/user/hduser/sales4")
>>
>> However, I want use this approach without creating table explicitly
>> myself with sqlContext etc
>>
>> This does not seem to work
>>
>> val c = sqlContext.setConf("spark.sql.parquet.compression.codec.",
>> "uncompressed")
>>
>> Can I do through a method on DataFrame "s" above to make the table saved
>> as uncompressed?
>>
>> Thanks,
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>
>

Mime
View raw message