spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Armbrust <mich...@databricks.com>
Subject Re: SQLContext.applySchema strictness
Date Sun, 15 Feb 2015 01:30:11 GMT
Applying schema is a pretty low-level operation, and I would expect most
users would use the type safe interfaces.  If you are unsure you can always
run:

import org.apache.spark.sql.execution.debug._
schemaRDD.typeCheck()

and it will tell you if you have made any mistakes.

Michael

On Sat, Feb 14, 2015 at 1:05 PM, Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:

> Would it make sense to add an optional validate parameter to applySchema()
> which defaults to False, both to give users the option to check the schema
> immediately and to make the default behavior clearer?
> ‚Äč
>
> On Sat Feb 14 2015 at 9:18:59 AM Michael Armbrust <michael@databricks.com>
> wrote:
>
>> Doing runtime type checking is very expensive, so we only do it when
>> necessary (i.e. you perform an operation like adding two columns together)
>>
>> On Sat, Feb 14, 2015 at 2:19 AM, nitin <nitin2goyal@gmail.com> wrote:
>>
>>> AFAIK, this is the expected behavior. You have to make sure that the
>>> schema
>>> matches the row. It won't give any error when you apply the schema as it
>>> doesn't validate the nature of data.
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/SQLContext-applySchema-strictness-tp21650p21653.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>

Mime
View raw message