Would it make sense to add an optional
validate parameter to
applySchema() which defaults to False, both to give users the option to check the schema immediately and to make the default behavior clearer?
Doing runtime type checking is very expensive, so we only do it when necessary (i.e. you perform an operation like adding two columns together)On Sat, Feb 14, 2015 at 2:19 AM, nitin <email@example.com> wrote:AFAIK, this is the expected behavior. You have to make sure that the schema
matches the row. It won't give any error when you apply the schema as it
doesn't validate the nature of data.
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SQLContext-applySchema-strictness-tp21650p21653.html