spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicholas Chammas <nicholas.cham...@gmail.com>
Subject Re: SQLContext.applySchema strictness
Date Sat, 14 Feb 2015 21:05:16 GMT
Would it make sense to add an optional validate parameter to applySchema()
which defaults to False, both to give users the option to check the schema
immediately and to make the default behavior clearer?
‚Äč

On Sat Feb 14 2015 at 9:18:59 AM Michael Armbrust <michael@databricks.com>
wrote:

> Doing runtime type checking is very expensive, so we only do it when
> necessary (i.e. you perform an operation like adding two columns together)
>
> On Sat, Feb 14, 2015 at 2:19 AM, nitin <nitin2goyal@gmail.com> wrote:
>
>> AFAIK, this is the expected behavior. You have to make sure that the
>> schema
>> matches the row. It won't give any error when you apply the schema as it
>> doesn't validate the nature of data.
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/SQLContext-applySchema-strictness-tp21650p21653.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>

Mime
View raw message