spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alessandro Baretta <>
Subject SparkSQL not honoring schema
Date Thu, 11 Dec 2014 02:19:03 GMT

I defined a SchemaRDD by applying a hand-crafted StructType to an RDD. Some
of the Rows in the RDD are malformed--that is, they do not conform to the
schema defined by the StructType. When running a select statement on this
SchemaRDD I would expect SparkSQL to either reject the malformed rows or
fail. Instead, it returns whatever data it finds, even if malformed. Is
this the desired behavior? Is there no method in SparkSQL to check for
validity with respect to the schema?



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message