spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexander Hagerf (Jira)" <>
Subject [jira] [Created] (SPARK-29427) Create KeyValueGroupedDataset from RelationalGroupedDataset
Date Thu, 10 Oct 2019 10:36:00 GMT
Alexander Hagerf created SPARK-29427:

             Summary: Create KeyValueGroupedDataset from RelationalGroupedDataset
                 Key: SPARK-29427
             Project: Spark
          Issue Type: New Feature
          Components: Spark Core, SQL
    Affects Versions: 2.4.4
            Reporter: Alexander Hagerf

The scenario I'm having is that I'm reading two huge bucketed tables and since a regular join
is not performant enough for these cases I'm using groupByKey to generate two KeyValueGroupedDatasets
and cogroup them to implement the logic I need.

The issue with this approach is that I'm only grouping by the column that the tables are bucketed
by but since I'm using groupByKey the bucketing is completely ignored and I still get a full
What I'm looking for is some functionality to tell Catalyst to group by a column in a relational
way but then give the user a possibility to utilize the functions of the KeyValueGroupedDataset
e.g. cogroup (which is not available for dataframes)


At current spark (2.4.4) I see no way to do this efficiently. I think this is a valid use
case which if solved would have huge performance benefits.

This message was sent by Atlassian Jira

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message