spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cheng Lian <lian.cs....@gmail.com>
Subject Re: Lazy casting with Catalyst
Date Sat, 28 Mar 2015 15:35:13 GMT
Hi Pat,

I don't understand what "lazy casting" mean here. Why do you think 
current Catalyst casting is "eager"? Casting happens at runtime, and 
doesn't disable column pruning.

Cheng

On 3/28/15 11:26 PM, Patrick Woody wrote:
> Hi all,
>
> In my application, we take input from Parquet files where BigDecimals are
> written as Strings to maintain arbitrary precision.
>
> I was hoping to convert these back over to Decimal with Unlimited
> precision, but I'd still like to maintain the Parquet column pruning (all
> my attempts thus far seem to bring in the whole Row). Is it possible to do
> this lazily through catalyst?
>
> Basically I'd want to do Cast(col, DecimalType()) whenever col is actually
> referenced. Any tips on how to approach this would be appreciated.
>
> Thanks!
> -Pat
>


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


Mime
View raw message