Use a format that has built-in indexes, such as Parquet or Orc. Do not forget to sort the data on the columns that your filter on.

On 14 Aug 2016, at 05:03, Taotao.Li <charles.upboy@gmail.com> wrote:


hi, guys, does Spark SQL support indexes?  if so, how can I create an index on my temp table? if not, how can I handle some specific queries on a very large table? it would iterate all the table even though all I want is just a small piece of that table.

great thanks, 


___________________
Quant | Engineer | Boy
___________________
blog:    http://litaotao.github.io
githubwww.github.com/litaotao