spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cheng Lian <>
Subject Re: How to disable parquet schema merging in 1.4?
Date Wed, 01 Jul 2015 20:13:28 GMT
With Spark 1.4, you may use data source option "mergeSchema" to control it:"mergeSchema", "false").parquet("some/path")


   CREATE TABLE t USING parquet OPTIONS (mergeSchema false, path 

We're considering to disable schema merging by default in 1.5.0 since it 
brings unnecessary performance cost when schema evolution is not a problem.


On 6/23/15 2:20 AM, Rex Xiong wrote:
> I remember in a previous PR, schema merging can be disabled by 
> setting spark.sql.hive.convertMetastoreParquet.mergeSchema to false.
> But in 1.4 release, I don't see this config anymore, is there a new 
> way to do it?
> Thanks

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message