samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Julian Hyde <jul...@hydromatic.net>
Subject Re: [DISCUSS] SQL workflow
Date Wed, 11 Feb 2015 18:51:07 GMT
This seems more like a branch than a classifier. Calcite is developing in a branch, and would
produce snapshots from that branch. The rule of thumb I’ve learned integrating Calcite into
Hive is that only a branch should depend on snapshot versions of another component. (Hive
broke this rule and got into a sticky situation where their trunk depended on a Calcite snapshot
and they couldn’t release because Calcite was stuck in pre-release vote limbo.)

I’m going to try to produce maven artifacts with {groupId: “org.apache.calcite”, version:
"1.1.0-stream-incubating-SNAPSHOT”, artifactId: “calcite-core”, “calcite-avatica”,
etc.} and push them to Apache nexus. I do recommend that you use it on a branch.

However, I don’t think there would be a problem getting the functionality into calcite’s
master branch in time for Calcite’s next release, in about 4 weeks. (We commit to releasing
approximately monthly.) The functionality and APIs would be marked “experimental, subject
to change” but at least you would be able to merge your work to master at that point. Doubtless
you would need new features & bug fixes in Calcite after that, and so work that depended
on that would have to stay in a branch until Calcite was released.

Julian

On Feb 11, 2015, at 10:34 AM, Felix GV <fvillegas@linkedin.com.INVALID> wrote:

> I guess one concern with Calcite SNAPSHOT releases is that they imply they'll be included
in the next release, which may not be the case if they're still considered very experimental.
> 
> Alternatively, perhaps Julian can create a new version name with a classifier for streaming.
> 
> http://maven.apache.org/plugins/maven-deploy-plugin/examples/deploying-with-classifiers.html
> 
> Then he could release SNAPSHOT artifacts for both the main/master code base as well as
the streaming branch, no matter how experimental it is...
> 
> --
> 
> Felix GV
> Data Infrastructure Engineer
> Distributed Data Systems
> LinkedIn
> 
> fgv@linkedin.com
> linkedin.com/in/felixgv
> 
> ________________________________________
> From: Chris Riccomini [criccomini@apache.org]
> Sent: Wednesday, February 11, 2015 10:23 AM
> To: Chris Riccomini
> Cc: dev@samza.apache.org
> Subject: Re: [DISCUSS] SQL workflow
> 
> Hey Julian,
> 
> It looks like Calcite is at least setup to publish to Maven:
> 
>  https://repository.apache.org/#nexus-search;quick~calcite
> 
> Julian, can you publish SNAPSHOT releases with your streaming changes to it?
> 
> Thanks!
> Chris
> 
> On Wed, Feb 11, 2015 at 10:08 AM, Chris Riccomini <criccomini@apache.org>
> wrote:
> 
>> Hey Milinda,
>> 
>> I've just committed SAMZA-484, so you should be able to re-base and get
>> all the latest code.
>> 
>>> But we need to add Calcite as a dependency and Calcite streaming
>> support is only in Julian's branch
>> 
>> We'll have to have Julian publish a release to Apache. This can be a
>> SNAPSHOT to Apache's SNAPSHOT repo, or a real Calcite release. We can't
>> check in binaries to the repo (licensing, release complexity, and just
>> generally a bad idea). We'll have to coordinate with him.
>> 
>> Cheers,
>> Chris
>> 
>> On Wed, Feb 11, 2015 at 9:43 AM, Milinda Pathirage <mpathira@umail.iu.edu>
>> wrote:
>> 
>>> No, SAMZA-483 doesn't depend on SAMZA-484. But we need to add Calcite as a
>>> dependency and Calcite streaming support is only in Julian's branch (
>>> https://github.com/julianhyde/incubator-calcite/tree/chi). What can we do
>>> about that?
>>> 
>>> Milinda
>>> 
>>> On Wed, Feb 11, 2015 at 12:30 PM, Chris Riccomini <criccomini@apache.org>
>>> wrote:
>>> 
>>>> Hey Milinda,
>>>> 
>>>> That'd be great. Do you have any dependency on SAMZA-484? I haven't had
>>> a
>>>> chance to commit that one yet.
>>>> 
>>>> Cheers,
>>>> Chris
>>>> 
>>>> On Wed, Feb 11, 2015 at 9:23 AM, Milinda Pathirage <
>>> mpathira@umail.iu.edu>
>>>> wrote:
>>>> 
>>>>> /need/want
>>>>> 
>>>>> Milinda
>>>>> 
>>>>> On Wed, Feb 11, 2015 at 12:22 PM, Milinda Pathirage <
>>>> mpathira@umail.iu.edu
>>>>>> 
>>>>> wrote:
>>>>> 
>>>>>> Hi Chris,
>>>>>> 
>>>>>> Do you need me to create the SAMZA-483 patch against latest master
>>> with
>>>>>> SAMZA-482 patch? I think that will make it easier to review the
>>> patch?
>>>>>> 
>>>>>> Thanks
>>>>>> Milinda
>>>>>> 
>>>>>> On Mon, Feb 9, 2015 at 11:39 AM, Chris Riccomini <
>>>> criccomini@apache.org>
>>>>>> wrote:
>>>>>> 
>>>>>>> Cool. Looks like we've got consensus. I'll move forward with
RTC on
>>>> some
>>>>>>> of
>>>>>>> the early SAMZA-390 tickets. (SAMZA-482, SAMZA-483, SAMZA-484)
>>>>>>> 
>>>>>>> On Mon, Feb 9, 2015 at 8:35 AM, Yan Fang <yanfang724@gmail.com>
>>>> wrote:
>>>>>>> 
>>>>>>>> +1 on this.
>>>>>>>> 
>>>>>>>> Fang, Yan
>>>>>>>> yanfang724@gmail.com
>>>>>>>> +1 (206) 849-4108
>>>>>>>> 
>>>>>>>> On Fri, Feb 6, 2015 at 4:38 PM, Chris Riccomini <
>>>>> criccomini@apache.org>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hey all,
>>>>>>>>> 
>>>>>>>>> Are we +1 on this? I think Jakob was the only one who
was
>>> curious
>>>>>>> about
>>>>>>>> it.
>>>>>>>>> 
>>>>>>>>> Cheers,
>>>>>>>>> Chris
>>>>>>>>> 
>>>>>>>>> On Thu, Feb 5, 2015 at 1:22 PM, Yi Pan <nickpan47@gmail.com>
>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hi, Jakob,
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>>> Eh? Not sure what this means...
>>>>>>>>>>> 
>>>>>>>>>>> I mean SAMZA-484 depends on SAMZA-482, and neither
are
>>>>> committed.
>>>>>>> So
>>>>>>>>>> Navina
>>>>>>>>>>> is having to post Yi's patch, as well as her
own, on the
>>> JIRA.
>>>>> It
>>>>>>>> makes
>>>>>>>>>> it
>>>>>>>>>>> really hard to do code reviews because you can't
tell
>>> whether
>>>> Yi
>>>>>>> made
>>>>>>>>> the
>>>>>>>>>>> changes or Navina did.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> Just to add to the point. It is also difficult to
always see
>>> a
>>>>> long
>>>>>>>> list
>>>>>>>>> of
>>>>>>>>>> changed files if the RB request is always based on
the
>>> master.
>>>> It
>>>>> is
>>>>>>>>>> possible to have RB request based on another RB request
(I
>>> have
>>>>>>> tried
>>>>>>>> it
>>>>>>>>>> before). But what happens if the base RB request
is
>>>>>>>> cancelled/discarded?
>>>>>>>>> RB
>>>>>>>>>> is not designed to track the revision changes in
a dependency
>>>>> chain.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Chris
>>>>>>>>>>> 
>>>>>>>>>>> On Thu, Feb 5, 2015 at 12:16 PM, Jakob Homan
<
>>>> jghoman@gmail.com
>>>>>> 
>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>>> I want to avoid branches,
>>>>>>>>>>>> Just curious, any reason for this?
>>>>>>>>>>>> 
>>>>>>>>>>>>> and I also want to avoid revision control
over JIRA
>>>>>>>>>>>> Eh? Not sure what this means...
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> jg
>>>>>>>>>>>> 
>>>>>>>>>>>> On 4 February 2015 at 17:11, Chris Riccomini
<
>>>>>>>> criccomini@apache.org>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> @Jakob, yeah I was thinking we'll follow
our normal
>>> flow.
>>>>>>> RTC. I
>>>>>>>>> just
>>>>>>>>>>>>> wanted to set expectation that the code
committed
>>> might be
>>>>>>> not up
>>>>>>>>> to
>>>>>>>>>>> our
>>>>>>>>>>>>> normal quality initially (missing docs,
no tests, etc).
>>>>> Until
>>>>>>> the
>>>>>>>>>>> quality
>>>>>>>>>>>>> is raised, we should think of this module
as
>>> experimental.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> @Milinda, awesome! Thanks. :)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Chris
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Wed, Feb 4, 2015 at 11:57 AM, Milinda
Pathirage <
>>>>>>>>>>>> mpathira@umail.iu.edu>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi Chris,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hope we no longer need the SQL API.
I'll create a RB
>>> for
>>>>>>> Calcite
>>>>>>>>>>>>>> integration.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>> Milinda
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Wed, Feb 4, 2015 at 1:31 PM, Chris
Riccomini <
>>>>>>>>>>> criccomini@apache.org>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I think so. There was some RB
downtime, but it just
>>> got
>>>>>>> fixed.
>>>>>>>>> Yi,
>>>>>>>>>>>>>> Navina,
>>>>>>>>>>>>>>> Milinda, can you make sure your
JIRAs have up to
>>> date
>>>>> RBs?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Wed, Feb 4, 2015 at 10:24
AM, sriram <
>>>>>>> sriram.sub@gmail.com
>>>>>>>>> 
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Can we have updated RBs for
all the three sub
>>> tasks
>>>>>>> before
>>>>>>>> we
>>>>>>>>>>>> commit?
>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>> would help us to review even
after we commit.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Wed, Feb 4, 2015 at 10:15
AM, Chris Riccomini <
>>>>>>>>>>>>>> criccomini@apache.org>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Yi, Navina, and Milinda
have been working on
>>>>> SAMZA-390
>>>>>>>>>>> sub-tickets
>>>>>>>>>>>>>>>> related
>>>>>>>>>>>>>>>>> to SQL operators. We're
getting to the point
>>> where
>>>>> the
>>>>>>>>> amount
>>>>>>>>>> of
>>>>>>>>>>>> work
>>>>>>>>>>>>>>>>> floating around is quite
large, and some tickets
>>>>> build
>>>>>>> off
>>>>>>>>> of
>>>>>>>>>>>> others.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I'm proposing that we
commit this work into a
>>>>> samza-sql
>>>>>>>>>>> submodule
>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>> master, and treat this
module as experimental. I
>>>> want
>>>>>>> to
>>>>>>>>> avoid
>>>>>>>>>>>>>>> branches,
>>>>>>>>>>>>>>>>> and I also want to avoid
revision control over
>>>> JIRA.
>>>>>>> This
>>>>>>>>>> means
>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>> will probably be a fair
amount of commits/JIRAs
>>> on
>>>>> this
>>>>>>>>> module
>>>>>>>>>>> as
>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>> iterate, but I think
that's OK.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Does this sound good
to everyone?
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Milinda Pathirage
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> PhD Student | Research Assistant
>>>>>>>>>>>>>> School of Informatics and Computing
| Data to Insight
>>>>> Center
>>>>>>>>>>>>>> Indiana University
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> twitter: milindalakmal
>>>>>>>>>>>>>> skype: milinda.pathirage
>>>>>>>>>>>>>> blog: http://milinda.pathirage.org
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Milinda Pathirage
>>>>>> 
>>>>>> PhD Student | Research Assistant
>>>>>> School of Informatics and Computing | Data to Insight Center
>>>>>> Indiana University
>>>>>> 
>>>>>> twitter: milindalakmal
>>>>>> skype: milinda.pathirage
>>>>>> blog: http://milinda.pathirage.org
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Milinda Pathirage
>>>>> 
>>>>> PhD Student | Research Assistant
>>>>> School of Informatics and Computing | Data to Insight Center
>>>>> Indiana University
>>>>> 
>>>>> twitter: milindalakmal
>>>>> skype: milinda.pathirage
>>>>> blog: http://milinda.pathirage.org
>>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Milinda Pathirage
>>> 
>>> PhD Student | Research Assistant
>>> School of Informatics and Computing | Data to Insight Center
>>> Indiana University
>>> 
>>> twitter: milindalakmal
>>> skype: milinda.pathirage
>>> blog: http://milinda.pathirage.org
>>> 
>> 
>> 


Mime
View raw message