spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cheng Lian <lian.cs....@gmail.com>
Subject Re: How to disable parquet schema merging in 1.4?
Date Wed, 01 Jul 2015 20:13:28 GMT
With Spark 1.4, you may use data source option "mergeSchema" to control it:

   sqlContext.read.option("mergeSchema", "false").parquet("some/path")

or

   CREATE TABLE t USING parquet OPTIONS (mergeSchema false, path 
"some/path")

We're considering to disable schema merging by default in 1.5.0 since it 
brings unnecessary performance cost when schema evolution is not a problem.

Cheng

On 6/23/15 2:20 AM, Rex Xiong wrote:
> I remember in a previous PR, schema merging can be disabled by 
> setting spark.sql.hive.convertMetastoreParquet.mergeSchema to false.
> But in 1.4 release, I don't see this config anymore, is there a new 
> way to do it?
>
> Thanks


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message