spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Armbrust <>
Subject Re: About Databricks's spark-sql-perf
Date Thu, 13 Aug 2015 18:13:32 GMT
Hey sorry, I've been doing a bunch of refactoring on this project.  Most of
the data generation was a huge hack (it was done before we supported
partitioning natively) and used some private APIs that don't exist
anymore.  As a result, while doing the regression tests for 1.5 I deleted a
bunch of broken code.  Those file should be deleted to.

The code does work with Spark 1.4/1.5, but at least as of today mostly
requires that you have already created the data/tables. I'll work on
updating the README as the QA period moves forward.

On Thu, Aug 13, 2015 at 6:49 AM, Todd <> wrote:

> Hi,
> I got a question about the spark-sql-perf project by Databricks at
> The Tables.scala (
> and BigData (
> are  empty files.
> Is this by intention or this is a bug.
> Also,the code snippet as follows in the README.MD won't compile  as there
> is no Tables class defined in the org.apache.spark.sql.parquet package:
> (I am using Spark1.4.1, is the code compatible with Spark 1.4.1?)
> import org.apache.spark.sql.parquet.Tables
> // Tables in TPC-DS benchmark used by experiments.
> val tables = Tables(sqlContext)
> // Setup TPC-DS experiment
> val tpcds = new TPCDS (sqlContext = sqlContext)

View raw message