spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hyukjin Kwon (Jira)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-29575) from_json can produce nulls for fields which are marked as non-nullable
Date Mon, 28 Oct 2019 12:30:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-29575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Hyukjin Kwon updated SPARK-29575:
---------------------------------
    Component/s: SQL

> from_json can produce nulls for fields which are marked as non-nullable
> -----------------------------------------------------------------------
>
>                 Key: SPARK-29575
>                 URL: https://issues.apache.org/jira/browse/SPARK-29575
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark, SQL
>    Affects Versions: 2.4.4
>            Reporter: Victor Lopez
>            Priority: Major
>
> I believe this issue was resolved elsewhere (https://issues.apache.org/jira/browse/SPARK-23173),
though for Pyspark this bug seems to still be there.
> The issue appears when using {{from_json}} to parse a column in a Spark dataframe. It
seems like {{from_json}} ignores whether the schema provided has any {{nullable:False}} property.
> {code:java}
> schema = T.StructType().add(T.StructField('id', T.LongType(), nullable=False)).add(T.StructField('name',
T.StringType(), nullable=False))
> data = [{'user': str({'name': 'joe', 'id':1})}, {'user': str({'name': 'jane'})}]
> df = spark.read.json(sc.parallelize(data))
> df.withColumn("details", F.from_json("user", schema)).select("details.*").show()
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message