spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adrian Bridgett <adr...@opensignal.com>
Subject Re: Issue with rogue data in csv file used in Spark application
Date Tue, 27 Sep 2016 21:53:30 GMT
We use the spark-csv (a successor of which is built in to spark 2.0) for 
this.  It doesn't cause crashes, failed parsing is logged.   We run on 
Mesos so I have to pull back all the logs from all the executors and 
search for failed lines (so that we can ensure that the failure rate 
isn't too high).

Hope this helps.

Adrian


---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message