spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Blue <rb...@netflix.com.INVALID>
Subject Re: [DISCUSS] preferred behavior when fails to instantiate configured v2 session catalog
Date Fri, 23 Oct 2020 17:02:41 GMT
I agree. If the user configures an invalid catalog, it should fail and
propagate the exception. Running with a catalog other than the one the user
requested is incorrect.

On Fri, Oct 23, 2020 at 5:24 AM Russell Spitzer <russell.spitzer@gmail.com>
wrote:

> I was convinced that we should probably just fail, but if that is too much
> of a change, then logging the exception is also acceptable.
>
> On Thu, Oct 22, 2020, 10:32 PM Jungtaek Lim <kabhwan.opensource@gmail.com>
> wrote:
>
>> Hi devs,
>>
>> I got another report regarding configuring v2 session catalog, when Spark
>> fails to instantiate the configured catalog. For now, it just simply logs
>> error message without exception information, and silently uses the default
>> session catalog.
>>
>>
>> https://github.com/apache/spark/blob/3819d39607392aa968595e3d97b84fedf83d08d9/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala#L75-L95
>>
>> IMO, as the user intentionally provides the session catalog, it shouldn't
>> fail back and just throw the exception. Otherwise (if we still want to do
>> the failback), we need to add the exception information in the error log
>> message at least.
>>
>> Would like to hear the voices.
>>
>> Thanks,
>> Jungtaek Lim (HeartSaVioR)
>>
>

-- 
Ryan Blue
Software Engineer
Netflix

Mime
View raw message