spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Armbrust <mich...@databricks.com>
Subject Re: Does Spark SQL support indexes?
Date Sun, 14 Aug 2016 16:29:59 GMT
Using df.write.partitionBy is similar to a coarse-grained, clustered index
in a traditional database.  You can't use it on temporary tables, but it
will let you efficiently select small parts of a much larger table.

On Sat, Aug 13, 2016 at 11:13 PM, Jörn Franke <jornfranke@gmail.com> wrote:

> Use a format that has built-in indexes, such as Parquet or Orc. Do not
> forget to sort the data on the columns that your filter on.
>
> On 14 Aug 2016, at 05:03, Taotao.Li <charles.upboy@gmail.com> wrote:
>
>
> hi, guys, does Spark SQL support indexes?  if so, how can I create an
> index on my temp table? if not, how can I handle some specific queries on a
> very large table? it would iterate all the table even though all I want is
> just a small piece of that table.
>
> great thanks,
>
>
> *___________________*
> Quant | Engineer | Boy
> *___________________*
> *blog*:    http://litaotao.github.io
> <http://litaotao.github.io/?utm_source=spark_mail>
> *github*: www.github.com/litaotao
>
>
>

Mime
View raw message