spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeremy Freeman (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-4727) Add "dimensional" RDDs (time series, spatial)
Date Thu, 04 Dec 2014 15:22:12 GMT

    [ https://issues.apache.org/jira/browse/SPARK-4727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234269#comment-14234269
] 

Jeremy Freeman commented on SPARK-4727:
---------------------------------------

Great to brainstorm about this RJ! 

To some extent, we've been doing this over on the [Thunder|http://thefreemanlab.com/thunder/docs/]
project. In particular, check out the {{TimeSeries}} and {{Images}} classes [here|https://github.com/freeman-lab/thunder/tree/master/python/thunder/rdds],
which are essentially wrappers for specialized RDDs. Our basic abstraction is RDDs of ndarrays
(1D for time series, 2D or 3D for images/volumes), with metadeta (lazily propagated) for things
like dimensionality and time base, coordinates embedded in keys, and useful methods on these
objects like the ones you menion (e.g. filtering, fourier, cross-correlation). We've also
worked on transformations between representations, for the common case of sequences of images
corresponding to different time points.

We haven't worked on custom partition strategies yet, I think that will be most important
for image tiles drawn from a much larger image. There's cool work ongoing for that in GeoTrellis,
see the [repo|https://github.com/geotrellis/geotrellis/tree/master/spark/src/main] and a [talk|http://spark-summit.org/2014/talk/geotrellis-adding-geospatial-capabilities-to-spark]
from Rob.

FWIW, when we started it seemed more appropriate to build this into a specialized library,
rather than Spark core. It's also something that benefits from using Python, due to a bevy
of existing libraries for  temporal and image data (though there are certainly analogs in
Java/Scala). But it would be great to probe the community for general interest in these kinds
of abstractions and methods.

> Add "dimensional" RDDs (time series, spatial)
> ---------------------------------------------
>
>                 Key: SPARK-4727
>                 URL: https://issues.apache.org/jira/browse/SPARK-4727
>             Project: Spark
>          Issue Type: Brainstorming
>          Components: Spark Core
>    Affects Versions: 1.1.0
>            Reporter: RJ Nowling
>
> Certain types of data (times series, spatial) can benefit from specialized RDDs.  I'd
like to open a discussion about this.
> For example, time series data should be ordered by time and would benefit from operations
like:
> * Subsampling (taking every n data points)
> * Signal processing (correlations, FFTs, filtering)
> * Windowing functions
> Spatial data benefits from ordering and partitioning along a 2D or 3D grid.  For example,
path finding algorithms can optimized by only comparing points within a set distance, which
can be computed more efficiently by partitioning data into a grid.
> Although the operations on time series and spatial data may be different, there is some
commonality in the sense of the data having ordered dimensions and the implementations may
overlap.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message