spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arkadiusz Bicz <arkadiusz.b...@gmail.com>
Subject Re: Support for decimal separator (comma or period) in spark 2.1
Date Thu, 23 Feb 2017 12:53:43 GMT
Thank you Sam for answer, I have solved problem by loading all decimals
columns as string and replacing all commas with dots but this solution is
lacking of automatic infer schema which is quite nice functionality.

I can work on adding new option to DataFrameReader  for localization like:

spark.read.option("NumberLocale", "German").csv("filefromeurope.csv")

Just wonder if it will be accepted?

Best Regards,

Arkadiusz Bicz

On Thu, Feb 23, 2017 at 1:35 PM, Sam Elamin <hussam.elamin@gmail.com> wrote:

> Hi Arkadiuz
>
> Not sure if there is a localisation ability but I'm sure other will
> correct me if I'm wrong
>
>
> What you could do is write a udf function that replaces the commas with a .
>
> Assuming you know the column in question
>
>
> Regards
> Sam
> On Thu, 23 Feb 2017 at 12:31, Arkadiusz Bicz <arkadiusz.bicz@gmail.com>
> wrote:
>
>> Hi Team,
>>
>> I would like to know if it is possible to specify decimal localization
>> for DataFrameReader for  csv?
>>
>> I have cvs files from localization where decimal separator is comma like
>> 0,32 instead of US way like 0.32
>>
>> Is it a way to specify in current version of spark to provide
>> localization:
>>
>> spark.read.option("sep",";").option("header",
>> "true").option("inferSchema", "true").format("csv").load("
>> nonuslocalized.csv")
>>
>> If not should I create ticket in jira for this ? I can work on solution
>> if not available.
>>
>> Best Regards,
>>
>> Arkadiusz Bicz
>>
>>
>>

Mime
View raw message