spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Enrico Minack <m...@Enrico.Minack.dev>
Subject Re: Issue With mod function in Spark SQL
Date Wed, 18 Dec 2019 07:43:00 GMT
I think some example code would help to understand what you are doing.

Am 18.12.19 um 08:12 schrieb Tzahi File:
> no.. there're 100M records both even and odd
>
> On Tue, Dec 17, 2019 at 8:13 PM Russell Spitzer 
> <russell.spitzer@gmail.com <mailto:russell.spitzer@gmail.com>> wrote:
>
>     Is there a chance your data is all even or all odd?
>
>     On Tue, Dec 17, 2019 at 11:01 AM Tzahi File
>     <tzahi.file@ironsrc.com <mailto:tzahi.file@ironsrc.com>> wrote:
>
>         I have in my spark sql query a calculated field that gets the
>         value if field1 % 3.
>
>         I'm using this field as a partition so I expected to get 3
>         partitions in the mentioned case, and I do get. The issue
>         happened with even numbers (instead of 3 - 4,2 ... ).
>         When I tried to use even numbers, for example 4 I got only 2
>         partitions - 1 and 3.
>         Field 1 datatype is bigint.
>
>         Do you have any suggestions?
>
>
>         -- 
>         thanks,
>         Tzahi
>
>
>
> -- 
> Tzahi File
> Data Engineer
> ironSource <http://www.ironsrc.com/>
> email tzahi.file@ironsrc.com <mailto:tzahi.file@ironsrc.com>
> mobile +972-546864835 <tel:+972-546864835>
> fax +972-77-5448273
> ironSource HQ - 121 Derech Menachem Begin st. Tel Aviv
> ironsrc.com <http://www.ironsrc.com/>
> linkedin <https://www.linkedin.com/company/ironsource>twitter 
> <https://twitter.com/ironsource>facebook 
> <https://www.facebook.com/ironSource>googleplus 
> <https://plus.google.com/+ironsrc>
> This email (including any attachments) is for the sole use of the 
> intended recipient and may contain confidential information which may 
> be protected by legal privilege. If you are not the intended 
> recipient, or the employee or agent responsible for delivering it to 
> the intended recipient, you are hereby notified that any use, 
> dissemination, distribution or copying of this communication and/or 
> its content is strictly prohibited. If you are not the intended 
> recipient, please immediately notify us by reply email or by 
> telephone, delete this email and destroy any copies. Thank you.
>


Mime
View raw message