spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Segel <msegel_had...@hotmail.com>
Subject Re: Spark SQL Json Parse
Date Thu, 03 Mar 2016 18:50:51 GMT
Why do you want to write out NULL if the column has no data? 
Just insert the fields that you have. 


> On Mar 3, 2016, at 9:10 AM, barisak <baris.akgun1@gmail.com> wrote:
> 
> Hi,
> 
> I have a problem with Json Parser. I am using spark streaming with
> hiveContext for keeping json format tweets. The flume collects tweets and
> sink to hdfs path. My spark streaming job checks the hdfs path and convert
> coming json tweets and insert them to hive table.
> 
> My problem is that ;
> 
> Some of tweets have location name information or in reply to information
> etc. but some of tweets have not any information for that columns.
> 
> In Spark streaming job, I try to make generic json handler for example if
> the tweet has no location information spark can write null value to hive
> table for location column.
> 
> Is there any way to make this?
> 
> I am using Spark 1.3.0 version
> 
> Thanks.
> 
> 
> 
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-Json-Parse-tp26391.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message