spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexander Hagerf (Jira)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-29427) Create KeyValueGroupedDataset from RelationalGroupedDataset
Date Thu, 10 Oct 2019 10:38:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-29427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Alexander Hagerf updated SPARK-29427:
-------------------------------------
    Description: 
The scenario I'm having is that I'm reading two huge bucketed tables and since a regular join
is not performant enough for my cases, I'm using groupByKey to generate two KeyValueGroupedDatasets
and cogroup them to implement the merging logic I need.

The issue with this approach is that I'm only grouping by the column that the tables are bucketed
by but since I'm using groupByKey the bucketing is completely ignored and I still get a full
shuffle. 
 What I'm looking for is some functionality to tell Catalyst to group by a column in a relational
way but then give the user a possibility to utilize the functions of the KeyValueGroupedDataset
e.g. cogroup (which is not available for dataframes)

 

At current spark (2.4.4) I see no way to do this efficiently. I think this is a valid use
case which if solved would have huge performance benefits.

  was:
The scenario I'm having is that I'm reading two huge bucketed tables and since a regular join
is not performant enough for these cases I'm using groupByKey to generate two KeyValueGroupedDatasets
and cogroup them to implement the logic I need.

The issue with this approach is that I'm only grouping by the column that the tables are bucketed
by but since I'm using groupByKey the bucketing is completely ignored and I still get a full
shuffle. 
What I'm looking for is some functionality to tell Catalyst to group by a column in a relational
way but then give the user a possibility to utilize the functions of the KeyValueGroupedDataset
e.g. cogroup (which is not available for dataframes)

 

At current spark (2.4.4) I see no way to do this efficiently. I think this is a valid use
case which if solved would have huge performance benefits.


> Create KeyValueGroupedDataset from RelationalGroupedDataset
> -----------------------------------------------------------
>
>                 Key: SPARK-29427
>                 URL: https://issues.apache.org/jira/browse/SPARK-29427
>             Project: Spark
>          Issue Type: New Feature
>          Components: Spark Core, SQL
>    Affects Versions: 2.4.4
>            Reporter: Alexander Hagerf
>            Priority: Major
>
> The scenario I'm having is that I'm reading two huge bucketed tables and since a regular
join is not performant enough for my cases, I'm using groupByKey to generate two KeyValueGroupedDatasets
and cogroup them to implement the merging logic I need.
> The issue with this approach is that I'm only grouping by the column that the tables
are bucketed by but since I'm using groupByKey the bucketing is completely ignored and I still
get a full shuffle. 
>  What I'm looking for is some functionality to tell Catalyst to group by a column in
a relational way but then give the user a possibility to utilize the functions of the KeyValueGroupedDataset
e.g. cogroup (which is not available for dataframes)
>  
> At current spark (2.4.4) I see no way to do this efficiently. I think this is a valid
use case which if solved would have huge performance benefits.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message