spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Davies Liu <dav...@databricks.com>
Subject Re: Query regarding infering data types in pyspark
Date Sat, 11 Apr 2015 06:26:08 GMT
What's the format you have in json file?

On Fri, Apr 10, 2015 at 6:57 PM, Suraj Shetiya <surajshetiya@gmail.com> wrote:
> Hi,
>
> In pyspark when if I read a json file using sqlcontext I find that the date
> field is not infered as date instead it is converted to string. And when I
> try to convert it to date using df.withColumn(df.DateCol.cast("timestamp"))
> it does not parse it successfuly and adds a null instead there. Should I
> use UDF to convert the date ? Is this expected behaviour (not throwing an
> error after failure to cast all fields)?
>
> --
> Regards,
> Suraj

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


Mime
View raw message