spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vatsal <>
Subject Reasoning behind fail safe behaviour of cast expression
Date Wed, 18 Mar 2020 12:05:02 GMT
I noticed that Spark handles CAST operation in fail-safe manner. i.e. if the
casting operation will fail for some record(s), Spark doesn't fail the
entire query instead it returns null data value for those failures. 

For example following query: 

Looking at the code it seems that this behavior is implemented
intentionally. As per my understanding, is this behavior is implemented to
avoid failing the entire query due to some outlier records. Is this
understanding correct?

Is there any dev group mail thread where I can find details discussion on
this decision? 

Also is there any configuration that can be used to make the query fail fast
instead of returning null values? Any future development is planned on this
similar line?

Sent from:

To unsubscribe e-mail:

View raw message