spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Armbrust <mich...@databricks.com>
Subject Re: Spark SQL Hive - merge small files
Date Wed, 05 Aug 2015 17:02:40 GMT
This feature isn't currently supported.

On Wed, Aug 5, 2015 at 8:43 AM, Brandon White <bwwinthehouse@gmail.com>
wrote:

> Hello,
>
> I would love to have hive merge the small files in my managed hive context
> after every query. Right now, I am setting the hive configuration in my
> Spark Job configuration but hive is not managing the files. Do I need to
> set the hive fields in around place? How do you set Hive configurations in
> Spark?
>
> Here is what I'd like to set
>
> hive.merge.mapfilestrue
> hive.merge.mapredfilestrue
> hive.merge.size.per.task256000000
> hive.merge.smallfiles.avgsize16000000
>

Mime
View raw message