spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <so...@cloudera.com>
Subject Re: Catalyst dependency on Spark Core
Date Tue, 15 Jul 2014 10:03:21 GMT
Agree. You end up with a "core" and a "corer core" to distinguish
between and it ends up just being more complicated. This sounds like
something that doesn't need a module.

On Tue, Jul 15, 2014 at 5:59 AM, Patrick Wendell <pwendell@gmail.com> wrote:
> Adding new build modules is pretty high overhead, so if this is a case
> where a small amount of duplicated code could get rid of the
> dependency, that could also be a good short-term option.
>
> - Patrick
>
> On Mon, Jul 14, 2014 at 2:15 PM, Matei Zaharia <matei.zaharia@gmail.com> wrote:
>> Yeah, I'd just add a spark-util that has these things.
>>
>> Matei
>>
>> On Jul 14, 2014, at 1:04 PM, Michael Armbrust <michael@databricks.com>
>> wrote:
>>
>> Yeah, sadly this dependency was introduced when someone consolidated the
>> logging infrastructure.  However, the dependency should be very small and
>> thus easy to remove, and I would like catalyst to be usable outside of
>> Spark.  A pull request to make this possible would be welcome.
>>
>> Ideally, we'd create some sort of spark common package that has things like
>> logging.  That way catalyst could depend on that, without pulling in all of
>> Hadoop, etc.  Maybe others have opinions though, so I'm cc-ing the dev list.
>>
>>
>> On Mon, Jul 14, 2014 at 12:21 AM, Yanbo Liang <yanbohappy@gmail.com> wrote:
>>>
>>> Make Catalyst independent of Spark is the goal of Catalyst, maybe need
>>> time and evolution.
>>> I awared that package org.apache.spark.sql.catalyst.util embraced
>>> org.apache.spark.util.{Utils => SparkUtils},
>>> so that Catalyst has a dependency on Spark core.
>>> I'm not sure whether it will be replaced by other component independent of
>>> Spark in later release.
>>>
>>>
>>> 2014-07-14 11:51 GMT+08:00 Aniket Bhatnagar <aniket.bhatnagar@gmail.com>:
>>>
>>>> As per the recent presentation given in Scala days
>>>> (http://people.apache.org/~marmbrus/talks/SparkSQLScalaDays2014.pdf), it
was
>>>> mentioned that Catalyst is independent of Spark. But on inspecting pom.xml
>>>> of sql/catalyst module, it seems it has a dependency on Spark Core. Any
>>>> particular reason for the dependency? I would love to use Catalyst outside
>>>> Spark
>>>>
>>>> (reposted as previous email bounced. Sorry if this is a duplicate).
>>>
>>>
>>
>>

Mime
View raw message