spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tobias Pfeiffer <...@preferred.jp>
Subject Re: MatchError in JsonRDD.toLong
Date Tue, 20 Jan 2015 01:25:38 GMT
Hi,

On Fri, Jan 16, 2015 at 6:14 PM, Wang, Daoyuan <daoyuan.wang@intel.com>
wrote:

> The second parameter of jsonRDD is the sampling ratio when we infer schema.
>

OK, I was aware of this, but I guess I understand the problem now. My
sampling ratio is so low that I only see the Long values of data items and
infer it's a Long. When I meet the data that's actually longer than Long, I
get the error I posted; basically it's the same situation as when
specifying a wrong schema manually.

So is there any way around this other than increasing the sample ratio to
discover also the very BigDecimal-sized numbers?

Thanks
Tobias

Mime
View raw message