spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: Parquet files are only 6-20MB in size?
Date Mon, 03 Nov 2014 08:03:53 GMT
Before doing saveAsParquetFile, you can call the repartition and provide a
decent number which will result in the total number of output files
generated.

Thanks
Best Regards

On Mon, Nov 3, 2014 at 1:12 PM, ag007 <agrealy@mac.com> wrote:

> Hi there,
>
> I have a pySpark job that is simply taking a tab separated CSV outputting
> it
> to a Parquet file.  The code is based on the SQL write parquet example.
> (Using a different inferred schema, only 35 columns). The input files range
> from 100MB to 12 Gb.
>
> I have tried different different block sizes from 10MB through to 1 Gb, I
> have tried different parallelism. The total part files total about 1:5
> compression.
>
> I am trying to get large parquet files.  Having this many small files will
> cause problems to my name node.  I have over 500,000 of these files.
>
> Your assistance would be greatly appreciated.
>
> cheers,
> Ag
>
> PS Another solution may be if there is a parquet concat tool around.  I
> couldn't see one.  I understand that this tool would have to adjust the
> footer.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Parquet-files-are-only-6-20MB-in-size-tp17935.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message