spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin Burton <bur...@spinn3r.com>
Subject Re: status of spark analytics functions? over, rank, percentile, row_number, etc.
Date Mon, 12 Jan 2015 20:10:14 GMT
Great. I’d love to help out. Is there any documentation on what you’re
working on that I can take a look at?

My biggest issue is that I need some way to compute the position of an
entry when used by ORDER BY… which I can do with the RANK operator.

What I essentially need is:

select source, indegree, rank() over (order by in degree desc) from foo
order by indegree desc,

This would give me the position of the record in the whole index and the
table sorted by indegree desc.

I was using RANK in pig but we’re ditching hadoop/pig in favor of spark.

I assume you’re implementing something similar to this:

https://issues.apache.org/jira/browse/SPARK-1442

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+WindowingAndAnalytics

https://issues.apache.org/jira/browse/HIVE-4197

https://issues.apache.org/jira/browse/HIVE-896

On Sat, Jan 10, 2015 at 5:00 PM, Will Benton <willb@redhat.com> wrote:

> Hi Kevin,
>
> I'm currently working on implementing windowing.  If you'd like to see
> something that's not covered by a JIRA, please file one!
>
>
> best,
> wb
>
> ----- Original Message -----
> > From: "Kevin Burton" <burton@spinn3r.com>
> > To: user@spark.apache.org
> > Sent: Saturday, January 10, 2015 12:12:38 PM
> > Subject: status of spark analytics functions? over, rank, percentile,
> row_number, etc.
> >
> > I’m curious what the status of implementing hive analytics functions in
> > spark.
> >
> >
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+WindowingAndAnalytics
> >
> > Many of these seem missing.  I’m assuming they’re not implemented yet?
> >
> > Is there an ETA on them?
> >
> > or am I the first to bring this up? :-P
> >
> > --
> >
> > Founder/CEO Spinn3r.com
> > Location: *San Francisco, CA*
> > blog: http://burtonator.wordpress.com
> > … or check out my Google+ profile
> > <https://plus.google.com/102718274791889610666/posts>
> > <http://spinn3r.com>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>


-- 

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>

Mime
View raw message