spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Haviv <danielru...@gmail.com>
Subject Re: Spark SQL Exception handling
Date Thu, 20 Nov 2014 18:55:54 GMT
Thanks I'll try it out!

On Thu, Nov 20, 2014 at 8:39 PM, Michael Armbrust <michael@databricks.com>
wrote:

> If you run master or the 1.2 preview release then it should automatically
> skip lines that fail to parse.  The corrupted text will be in the column
> _corrupted_record and the other columns will be null.
>
> On Thu, Nov 20, 2014 at 7:34 AM, Daniel Haviv <danielrulez@gmail.com>
> wrote:
>
>> Hi Guys,
>> I really need your help with this:
>>
>> I'm loading a bunch of json files uploaded via webhdfs, some of them have
>> some incosistencies (the json ends mid-line for example) and that causes my
>> whole application to fail.
>> How can I continue processing valid json records without failing the
>> application?
>>
>>
>> Thanks,
>> Daniel
>>
>
>

Mime
View raw message