spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Russell Spitzer <russell.spit...@gmail.com>
Subject Re: [DISCUSS] preferred behavior when fails to instantiate configured v2 session catalog
Date Fri, 23 Oct 2020 12:24:32 GMT
I was convinced that we should probably just fail, but if that is too much
of a change, then logging the exception is also acceptable.

On Thu, Oct 22, 2020, 10:32 PM Jungtaek Lim <kabhwan.opensource@gmail.com>
wrote:

> Hi devs,
>
> I got another report regarding configuring v2 session catalog, when Spark
> fails to instantiate the configured catalog. For now, it just simply logs
> error message without exception information, and silently uses the default
> session catalog.
>
>
> https://github.com/apache/spark/blob/3819d39607392aa968595e3d97b84fedf83d08d9/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala#L75-L95
>
> IMO, as the user intentionally provides the session catalog, it shouldn't
> fail back and just throw the exception. Otherwise (if we still want to do
> the failback), we need to add the exception information in the error log
> message at least.
>
> Would like to hear the voices.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>

Mime
View raw message