calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <>
Subject Re: [DISCUSS] The state of the project
Date Tue, 04 Oct 2016 23:57:40 GMT

Julian Hyde wrote:
> Hi Calcite community members,
> In a few weeks (22nd October) it will be a year since Calcite graduated to a top-level
Apache project[1]. I think it’s been a good year!
> When we graduated, we decided to have an annual “state of the project” discussion
and to vote for a new PMC chair/VP[2]. So, I’m kicking off both of those discussions.
> First, a few of my thoughts.
> I am pleased with the general rate of progress of the project. I’m pleased to see an
increasing number of contributions from new contributors, and some of those becoming committers
and PMC members. A couple of highlights this year were adapters for Cassandra and Elasticsearch
that can out of the blue. I’m also pleased that we have continued a regular release cadence.
This makes it easier for projects to use Calcite, and knowing that pull requests will be promptly
reviewed and included in a release gives people an incentive to contribute.
> Calcite is becoming an ever better optimizer for SQL queries. This is helped immeasurably
by the fact that Hive, Phoenix, Drill, Qubole and others are using Calcite for this and are
contributing back. (Thanks to those communities for their continued collaboration!)

+1 this has been awesome to watch :)

> But I also believe that Calcite can be used for non-traditional databases. Some examples:
> 1. I am a fan of what Drill have done with schema-less query processing and document-oriented
data, and would like to bring similar functionality into core Calcite.

I remember I saw a presentation by someone on Drill a while back (very 
much a "intro to drill" by someone not affiliated with Calcite either). 
The way the content was presented it was so very clear the influences of 
Calcite into their architecture. Very cool to see!

> 2. I also like the idea of Calcite being a “toolkit” from which one can build a database
(relational or non-relational). Phoenix have been going through the process of converting
their existing parser&  planner to use Calcite, and I have learned a lot. But a lot still
needs to be done to make Calcite easier to use as a framework.
> 3. I have been building consensus that SQL is a great language for stream processing[3],
and working with Apex, Flink, Samza, Storm to build the pieces to implement streaming SQL.
I am very excited about the way streaming SQL is gaining acceptance. Are there any other emerging
areas should Calcite be targeting?
> Avatica continues to grow and mature. The Avatica site now lists clients in 4 languages[4],
and there is also an ODBC driver (not open source)[5]. The “one repo, one community, two
web sites, two releases” strategy seems to be working adequately. But where do we see the
project going? Would it help if it had its own namespace (org.apache.avatica) or web site
(<>)? Might it be a top-level project

I think, in time, Avatica could easily grow into its own entity. I don't 
think we're there yet.

I will say that I think there's been a regular amount of confusion with 
Avatica and Calcite sharing a repository but not following the same 
versioning scheme. People seem to be a bit confused when I tell them 
that the two projects are not "attached" to one another (they are 
separate Maven projects).

I think pulling Avatica into its own repo would be good encouragement to 
being its own entity (as well as drawing the line between Calcite and 
Avatica codebases), but I think this is low-priority (as there are few 
of us doing Avatica work) and we need to do a better job at clearly 
stating what Avatica is/does and its API.

> Regarding community. Are we doing enough to reach out and bring new members into the
community? Some of us have given talks at conferences and meetups over the last 12 months.
Could we improve our geographical reach? Are there other things we could do to make the project
more welcoming to new contributors? Could we do more to reach out to women and other demographic
groups underrepresented in our community?
> What else are we doing well in the project? What are areas where we need to do better?

As you outlined, adoption across other projects has been great. What 
about adoption by users? I know the last time I tried to hack some SQL 
system together with Calcite (albeit, quite a while ago), I was left 
wondering what is "public API" (what are the classes I should use versus 
what are those that are "internal"). I think we still see a fair amount 
of requests for "hand-holding" as well. I'm not sure how we make this 
better (or if it's a best use of time -- the csv example goes very far 
already!). Just a comment.

> Lastly, since I agreed to step down as VP after 12 months, let’s start talking about
a replacement. Being PMC chair is a privilege and it has taught me a huge amount about how
Apache works. I think that Jesús Camacho Rodríguez could do an excellent job, if he is willing.
Which other candidates should we consider?

+1 to Jesús if he's amenable to it. He's been a pleasure to work with 
and I'd have no complaints. Also happy to entertain others who would 
like to step up (without volunteering them myself ;P)

> Please take some time to share your thoughts about the state of the project.
> Julian
> (VP Apache Calcite)
> [1]<>
> [2]<>
> [3]<>
> [4]<>
> [5]<>

View raw message