spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Takuya Ueshin (Jira)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-28962) High-order function: filter(array<T>, function<T, Int, boolean>) → array<T>
Date Wed, 02 Oct 2019 20:05:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-28962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Takuya Ueshin resolved SPARK-28962.
-----------------------------------
    Fix Version/s: 3.0.0
         Assignee: Henry Davidge
       Resolution: Fixed

Issue resolved by pull request 25666
https://github.com/apache/spark/pull/25666

> High-order function: filter(array<T>, function<T, Int, boolean>) → array<T>
> ---------------------------------------------------------------------------
>
>                 Key: SPARK-28962
>                 URL: https://issues.apache.org/jira/browse/SPARK-28962
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 3.0.0
>            Reporter: Henry Davidge
>            Assignee: Henry Davidge
>            Priority: Major
>             Fix For: 3.0.0
>
>
> It's helpful to have access to the index when using the {{filter}} function. For instance,
we're using SparkSQL to manipulate genomic data. We store some fields in a long array that
has the same length for every row in the DataFrame. We compute aggregates that are per array
position (so we look at the kth element for each row's array) and then want to filter each
row's array by looking values in the aggregate array.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message