spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <>
Subject Re: Catalyst dependency on Spark Core
Date Mon, 14 Jul 2014 21:15:44 GMT
Yeah, I'd just add a spark-util that has these things.


On Jul 14, 2014, at 1:04 PM, Michael Armbrust <> wrote:

> Yeah, sadly this dependency was introduced when someone consolidated the logging infrastructure.
 However, the dependency should be very small and thus easy to remove, and I would like catalyst
to be usable outside of Spark.  A pull request to make this possible would be welcome.
> Ideally, we'd create some sort of spark common package that has things like logging.
 That way catalyst could depend on that, without pulling in all of Hadoop, etc.  Maybe others
have opinions though, so I'm cc-ing the dev list.
> On Mon, Jul 14, 2014 at 12:21 AM, Yanbo Liang <> wrote:
> Make Catalyst independent of Spark is the goal of Catalyst, maybe need time and evolution.
> I awared that package org.apache.spark.sql.catalyst.util embraced org.apache.spark.util.{Utils
=> SparkUtils},
> so that Catalyst has a dependency on Spark core. 
> I'm not sure whether it will be replaced by other component independent of Spark in later
> 2014-07-14 11:51 GMT+08:00 Aniket Bhatnagar <>:
> As per the recent presentation given in Scala days (,
it was mentioned that Catalyst is independent of Spark. But on inspecting pom.xml of sql/catalyst
module, it seems it has a dependency on Spark Core. Any particular reason for the dependency?
I would love to use Catalyst outside Spark
> (reposted as previous email bounced. Sorry if this is a duplicate).

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message