spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cheng Lian <>
Subject Re: Lazy casting with Catalyst
Date Sat, 28 Mar 2015 15:35:13 GMT
Hi Pat,

I don't understand what "lazy casting" mean here. Why do you think 
current Catalyst casting is "eager"? Casting happens at runtime, and 
doesn't disable column pruning.


On 3/28/15 11:26 PM, Patrick Woody wrote:
> Hi all,
> In my application, we take input from Parquet files where BigDecimals are
> written as Strings to maintain arbitrary precision.
> I was hoping to convert these back over to Decimal with Unlimited
> precision, but I'd still like to maintain the Parquet column pruning (all
> my attempts thus far seem to bring in the whole Row). Is it possible to do
> this lazily through catalyst?
> Basically I'd want to do Cast(col, DecimalType()) whenever col is actually
> referenced. Any tips on how to approach this would be appreciated.
> Thanks!
> -Pat

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message