spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam Roberts (JIRA)" <>
Subject [jira] [Commented] (SPARK-4633) Support gzip in
Date Mon, 12 Sep 2016 16:00:23 GMT


Adam Roberts commented on SPARK-4633:

Very interested in this and I know Nasser Ebrahim is also (full disclosure that we both work
for IBM). shows promising results

Would be interesting to code up a quick prototype (perhaps based on the pull request here)
and to see what performance difference we can gain, looks like Takeshi has done the starting
work for us

> Support gzip in
> ------------------------------------------
>                 Key: SPARK-4633
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Input/Output
>            Reporter: Takeshi Yamamuro
>            Priority: Trivial
> gzip is widely used in other frameowrks such as hadoop mapreduce and tez, and also
> I think that gizip is more stable than other codecs in terms of both performance
> and space overheads.
> I have one open question; current spark configuratios have a block size option
> for each codec ([gzip|lz4|snappy].block.size).
> As # of codecs increases, the configurations have more options and
> I think that it is sort of complicated for non-expert users.
> To mitigate it, my thought follows;
> the three configurations are replaced with a single option for block size
> ( Then, 'Meaning' in configurations
> will describe "This option makes an effect on gzip, lz4, and snappy. 
> Block size (in bytes) used in compression, in the case when these compression
> codecs are used. Lowering...".

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message