spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Russell Spitzer <>
Subject Re: Sort order in bucketing in a custom datasource
Date Tue, 16 Apr 2019 23:01:07 GMT
Please join the DataSource V2 meetings, the next one is tomorrow since we
are discussing these very topics right now. Datasource v1 cannot provide
this information but any source which
just generates RDDs can specify a partitioner. This is only useful though
if you are only using RDDs, for Dataframes DSV2 is the place to look.

On Tue, Apr 16, 2019 at 5:31 PM Long, Andrew <>

> Hey Friends,
> Is it possible to specify the sort order or bucketing in a way that can be
> used by the optimizer in spark?
> Cheers Andrew

View raw message