spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rex Xiong <bycha...@gmail.com>
Subject Re: Issue of Hive parquet partitioned table schema mismatch
Date Sat, 31 Oct 2015 11:38:41 GMT
Add back this thread to email list, forgot to reply all.
2015年10月31日 下午7:23,"Michael Armbrust" <michael@databricks.com>写道:

> Not that I know of.
>
> On Sat, Oct 31, 2015 at 12:22 PM, Rex Xiong <bychance@gmail.com> wrote:
>
>> Good to know that, will have a try.
>> So there is no easy way to achieve it in pure hive method?
>> 2015年10月31日 下午7:17,"Michael Armbrust" <michael@databricks.com>写道:
>>
>>> Yeah, this was rewritten to be faster in Spark 1.5.  We use it with
>>> 10,000s of partitions.
>>>
>>> On Sat, Oct 31, 2015 at 7:17 AM, Rex Xiong <bychance@gmail.com> wrote:
>>>
>>>> 1.3.1
>>>> It is a lot of improvement in 1.5+?
>>>>
>>>> 2015-10-30 19:23 GMT+08:00 Michael Armbrust <michael@databricks.com>:
>>>>
>>>>> We have tried schema merging feature, but it's too slow, there're
>>>>>> hundreds of partitions.
>>>>>>
>>>>> Which version of Spark?
>>>>>
>>>>
>>>>
>>>
>

Mime
View raw message