spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrés Ivaldi <iaiva...@gmail.com>
Subject Re: Grouping Set
Date Thu, 17 Nov 2016 17:05:35 GMT
I'm realize that my data have null values, so that null are for the values
not for the calculated grouping set, but that is another problem, how can I
detect witch is one? now I have this problem

I my data is just a row like this [ {1:"A",2:null, 3:123}] the grouping set
(1) will give me
A, null, 123
A, null, 123

and with  [ {1:"A", 2:null, 3:123},{1:"A", 2:"b", 3:1}] the grouping set
(1) will give me
A, null, 124
A, null, 123
A, b, 1

Quick fix could be isNull with a label that I can detect, but that's too
dirty I think, grouping set should return a value type witch could be
detected as the grouped set on that column, not null


On Mon, Nov 14, 2016 at 5:49 PM, ayan guha <guha.ayan@gmail.com> wrote:

> And, run the same SQL in hive and post any difference.
> On 15 Nov 2016 07:48, "ayan guha" <guha.ayan@gmail.com> wrote:
>
>> It should be A,yes. Can you please reproduce this with small data and
>> exact SQL?
>> On 15 Nov 2016 02:21, "Andrés Ivaldi" <iaivaldi@gmail.com> wrote:
>>
>>> Hello, I'm tryin to use Grouping Set, but I dont know if it is a bug or
>>> the correct behavior.
>>>
>>> Givven the above example
>>> Select a,b,sum(c) from table group by a,b grouping set ( (a), (a,b) )
>>>
>>> What shound be the expected result
>>> A:
>>>
>>> A  | B    | sum(c)
>>> xx | null | ####
>>> xx | yy   | ####
>>> xx | zz   | ####
>>>
>>>
>>> B
>>> A   | B    | sum(c)
>>> xx  | null | ####
>>> xx  | yy   | ####
>>> xx  | zz   | ####
>>> null| yy   | ####
>>> null| zz   | ####
>>> null| null | ####
>>>
>>>
>>> I believe is A, but i'm getting B
>>> thanks
>>>
>>> --
>>> Ing. Ivaldi Andres
>>>
>>


-- 
Ing. Ivaldi Andres

Mime
View raw message