samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Julian Hyde <julianh...@gmail.com>
Subject Re: [DISCUSS] SQL workflow
Date Thu, 12 Feb 2015 18:56:30 GMT
+1

I've pushed a snapshot, versioned "1.1-chi-incubating-SNAPSHOT", and based on https://github.com/julianhyde/incubator-calcite/commit/b8a91c31859b6901976de8dcad43afd4877cad36.

See https://repository.apache.org/content/repositories/snapshots/org/apache/calcite/calcite/


> On Feb 12, 2015, at 9:26 AM, Jakob Homan <jghoman@gmail.com> wrote:
> 
> +1, assuming we're keeping RTC on the branch (, Jakob said pedantically).
> 
> On 12 February 2015 at 08:25, Milinda Pathirage <mpathira@umail.iu.edu> wrote:
>> I am +1 for this approach.
>> 
>> Milinda
>> 
>> On Thu, Feb 12, 2015 at 11:22 AM, Chris Riccomini <criccomini@apache.org>
>> wrote:
>> 
>>> Hey Julian,
>>> 
>>>> The rule of thumb I’ve learned integrating Calcite into Hive is that only
>>> a branch should depend on snapshot versions of another component
>>> 
>>> This is very practical. I hadn't considered the Calcite dependency issue
>>> when I pushed for merging into master rather than a branch.
>>> 
>>> Based on this, it seems like the best thing to do is:
>>> 
>>> 1) Work off of Calcite SNAPSHOT Maven publication.
>>> 2) Create a samza-sql branch.
>>> 3) Migrate all existing samza-sql commits from master into the samza-sql
>>> branch.
>>> 
>>> Sorry for the churn on this all. I hadn't considered the binary dependency
>>> issue. Is everyone OK with this?
>>> 
>>> Cheers,
>>> Chris
>>> 
>>> On Wed, Feb 11, 2015 at 10:51 AM, Julian Hyde <julian@hydromatic.net>
>>> wrote:
>>> 
>>>> This seems more like a branch than a classifier. Calcite is developing in
>>>> a branch, and would produce snapshots from that branch. The rule of thumb
>>>> I’ve learned integrating Calcite into Hive is that only a branch should
>>>> depend on snapshot versions of another component. (Hive broke this rule
>>> and
>>>> got into a sticky situation where their trunk depended on a Calcite
>>>> snapshot and they couldn’t release because Calcite was stuck in
>>> pre-release
>>>> vote limbo.)
>>>> 
>>>> I’m going to try to produce maven artifacts with {groupId:
>>>> “org.apache.calcite”, version: "1.1.0-stream-incubating-SNAPSHOT”,
>>>> artifactId: “calcite-core”, “calcite-avatica”, etc.} and push them
to
>>>> Apache nexus. I do recommend that you use it on a branch.
>>>> 
>>>> However, I don’t think there would be a problem getting the functionality
>>>> into calcite’s master branch in time for Calcite’s next release, in
>>> about 4
>>>> weeks. (We commit to releasing approximately monthly.) The functionality
>>>> and APIs would be marked “experimental, subject to change” but at least
>>> you
>>>> would be able to merge your work to master at that point. Doubtless you
>>>> would need new features & bug fixes in Calcite after that, and so work
>>> that
>>>> depended on that would have to stay in a branch until Calcite was
>>> released.
>>>> 
>>>> Julian
>>>> 
>>>> On Feb 11, 2015, at 10:34 AM, Felix GV <fvillegas@linkedin.com.INVALID>
>>>> wrote:
>>>> 
>>>>> I guess one concern with Calcite SNAPSHOT releases is that they imply
>>>> they'll be included in the next release, which may not be the case if
>>>> they're still considered very experimental.
>>>>> 
>>>>> Alternatively, perhaps Julian can create a new version name with a
>>>> classifier for streaming.
>>>>> 
>>>>> 
>>>> 
>>> http://maven.apache.org/plugins/maven-deploy-plugin/examples/deploying-with-classifiers.html
>>>>> 
>>>>> Then he could release SNAPSHOT artifacts for both the main/master code
>>>> base as well as the streaming branch, no matter how experimental it is...
>>>>> 
>>>>> --
>>>>> 
>>>>> Felix GV
>>>>> Data Infrastructure Engineer
>>>>> Distributed Data Systems
>>>>> LinkedIn
>>>>> 
>>>>> fgv@linkedin.com
>>>>> linkedin.com/in/felixgv
>>>>> 
>>>>> ________________________________________
>>>>> From: Chris Riccomini [criccomini@apache.org]
>>>>> Sent: Wednesday, February 11, 2015 10:23 AM
>>>>> To: Chris Riccomini
>>>>> Cc: dev@samza.apache.org
>>>>> Subject: Re: [DISCUSS] SQL workflow
>>>>> 
>>>>> Hey Julian,
>>>>> 
>>>>> It looks like Calcite is at least setup to publish to Maven:
>>>>> 
>>>>> https://repository.apache.org/#nexus-search;quick~calcite
>>>>> 
>>>>> Julian, can you publish SNAPSHOT releases with your streaming changes
>>> to
>>>> it?
>>>>> 
>>>>> Thanks!
>>>>> Chris
>>>>> 
>>>>> On Wed, Feb 11, 2015 at 10:08 AM, Chris Riccomini <
>>> criccomini@apache.org
>>>>> 
>>>>> wrote:
>>>>> 
>>>>>> Hey Milinda,
>>>>>> 
>>>>>> I've just committed SAMZA-484, so you should be able to re-base and
>>> get
>>>>>> all the latest code.
>>>>>> 
>>>>>>> But we need to add Calcite as a dependency and Calcite streaming
>>>>>> support is only in Julian's branch
>>>>>> 
>>>>>> We'll have to have Julian publish a release to Apache. This can be
a
>>>>>> SNAPSHOT to Apache's SNAPSHOT repo, or a real Calcite release. We
>>> can't
>>>>>> check in binaries to the repo (licensing, release complexity, and
just
>>>>>> generally a bad idea). We'll have to coordinate with him.
>>>>>> 
>>>>>> Cheers,
>>>>>> Chris
>>>>>> 
>>>>>> On Wed, Feb 11, 2015 at 9:43 AM, Milinda Pathirage <
>>>> mpathira@umail.iu.edu>
>>>>>> wrote:
>>>>>> 
>>>>>>> No, SAMZA-483 doesn't depend on SAMZA-484. But we need to add
Calcite
>>>> as a
>>>>>>> dependency and Calcite streaming support is only in Julian's
branch (
>>>>>>> https://github.com/julianhyde/incubator-calcite/tree/chi). What
can
>>>> we do
>>>>>>> about that?
>>>>>>> 
>>>>>>> Milinda
>>>>>>> 
>>>>>>> On Wed, Feb 11, 2015 at 12:30 PM, Chris Riccomini <
>>>> criccomini@apache.org>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Hey Milinda,
>>>>>>>> 
>>>>>>>> That'd be great. Do you have any dependency on SAMZA-484?
I haven't
>>>> had
>>>>>>> a
>>>>>>>> chance to commit that one yet.
>>>>>>>> 
>>>>>>>> Cheers,
>>>>>>>> Chris
>>>>>>>> 
>>>>>>>> On Wed, Feb 11, 2015 at 9:23 AM, Milinda Pathirage <
>>>>>>> mpathira@umail.iu.edu>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> /need/want
>>>>>>>>> 
>>>>>>>>> Milinda
>>>>>>>>> 
>>>>>>>>> On Wed, Feb 11, 2015 at 12:22 PM, Milinda Pathirage <
>>>>>>>> mpathira@umail.iu.edu
>>>>>>>>>> 
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hi Chris,
>>>>>>>>>> 
>>>>>>>>>> Do you need me to create the SAMZA-483 patch against
latest master
>>>>>>> with
>>>>>>>>>> SAMZA-482 patch? I think that will make it easier
to review the
>>>>>>> patch?
>>>>>>>>>> 
>>>>>>>>>> Thanks
>>>>>>>>>> Milinda
>>>>>>>>>> 
>>>>>>>>>> On Mon, Feb 9, 2015 at 11:39 AM, Chris Riccomini
<
>>>>>>>> criccomini@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Cool. Looks like we've got consensus. I'll move
forward with RTC
>>> on
>>>>>>>> some
>>>>>>>>>>> of
>>>>>>>>>>> the early SAMZA-390 tickets. (SAMZA-482, SAMZA-483,
SAMZA-484)
>>>>>>>>>>> 
>>>>>>>>>>> On Mon, Feb 9, 2015 at 8:35 AM, Yan Fang <yanfang724@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> +1 on this.
>>>>>>>>>>>> 
>>>>>>>>>>>> Fang, Yan
>>>>>>>>>>>> yanfang724@gmail.com
>>>>>>>>>>>> +1 (206) 849-4108
>>>>>>>>>>>> 
>>>>>>>>>>>> On Fri, Feb 6, 2015 at 4:38 PM, Chris Riccomini
<
>>>>>>>>> criccomini@apache.org>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Are we +1 on this? I think Jakob was
the only one who was
>>>>>>> curious
>>>>>>>>>>> about
>>>>>>>>>>>> it.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Chris
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Thu, Feb 5, 2015 at 1:22 PM, Yi Pan
<nickpan47@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi, Jakob,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Eh? Not sure what this means...
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I mean SAMZA-484 depends on SAMZA-482,
and neither are
>>>>>>>>> committed.
>>>>>>>>>>> So
>>>>>>>>>>>>>> Navina
>>>>>>>>>>>>>>> is having to post Yi's patch,
as well as her own, on the
>>>>>>> JIRA.
>>>>>>>>> It
>>>>>>>>>>>> makes
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>> really hard to do code reviews
because you can't tell
>>>>>>> whether
>>>>>>>> Yi
>>>>>>>>>>> made
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> changes or Navina did.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Just to add to the point. It is also
difficult to always see
>>>>>>> a
>>>>>>>>> long
>>>>>>>>>>>> list
>>>>>>>>>>>>> of
>>>>>>>>>>>>>> changed files if the RB request is
always based on the
>>>>>>> master.
>>>>>>>> It
>>>>>>>>> is
>>>>>>>>>>>>>> possible to have RB request based
on another RB request (I
>>>>>>> have
>>>>>>>>>>> tried
>>>>>>>>>>>> it
>>>>>>>>>>>>>> before). But what happens if the
base RB request is
>>>>>>>>>>>> cancelled/discarded?
>>>>>>>>>>>>> RB
>>>>>>>>>>>>>> is not designed to track the revision
changes in a dependency
>>>>>>>>> chain.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Thu, Feb 5, 2015 at 12:16
PM, Jakob Homan <
>>>>>>>> jghoman@gmail.com
>>>>>>>>>> 
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I want to avoid branches,
>>>>>>>>>>>>>>>> Just curious, any reason
for this?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> and I also want to avoid
revision control over JIRA
>>>>>>>>>>>>>>>> Eh? Not sure what this means...
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> jg
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On 4 February 2015 at 17:11,
Chris Riccomini <
>>>>>>>>>>>> criccomini@apache.org>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> @Jakob, yeah I was thinking
we'll follow our normal
>>>>>>> flow.
>>>>>>>>>>> RTC. I
>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>> wanted to set expectation
that the code committed
>>>>>>> might be
>>>>>>>>>>> not up
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> our
>>>>>>>>>>>>>>>>> normal quality initially
(missing docs, no tests, etc).
>>>>>>>>> Until
>>>>>>>>>>> the
>>>>>>>>>>>>>>> quality
>>>>>>>>>>>>>>>>> is raised, we should
think of this module as
>>>>>>> experimental.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> @Milinda, awesome! Thanks.
:)
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Wed, Feb 4, 2015 at
11:57 AM, Milinda Pathirage <
>>>>>>>>>>>>>>>> mpathira@umail.iu.edu>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hi Chris,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hope we no longer
need the SQL API. I'll create a RB
>>>>>>> for
>>>>>>>>>>> Calcite
>>>>>>>>>>>>>>>>>> integration.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>> Milinda
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Wed, Feb 4, 2015
at 1:31 PM, Chris Riccomini <
>>>>>>>>>>>>>>> criccomini@apache.org>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I think so. There
was some RB downtime, but it just
>>>>>>> got
>>>>>>>>>>> fixed.
>>>>>>>>>>>>> Yi,
>>>>>>>>>>>>>>>>>> Navina,
>>>>>>>>>>>>>>>>>>> Milinda, can
you make sure your JIRAs have up to
>>>>>>> date
>>>>>>>>> RBs?
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Wed, Feb 4,
2015 at 10:24 AM, sriram <
>>>>>>>>>>> sriram.sub@gmail.com
>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Can we have
updated RBs for all the three sub
>>>>>>> tasks
>>>>>>>>>>> before
>>>>>>>>>>>> we
>>>>>>>>>>>>>>>> commit?
>>>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>>>> would help
us to review even after we commit.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Wed, Feb
4, 2015 at 10:15 AM, Chris Riccomini <
>>>>>>>>>>>>>>>>>> criccomini@apache.org>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Hey all,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Yi, Navina,
and Milinda have been working on
>>>>>>>>> SAMZA-390
>>>>>>>>>>>>>>> sub-tickets
>>>>>>>>>>>>>>>>>>>> related
>>>>>>>>>>>>>>>>>>>>> to SQL
operators. We're getting to the point
>>>>>>> where
>>>>>>>>> the
>>>>>>>>>>>>> amount
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> work
>>>>>>>>>>>>>>>>>>>>> floating
around is quite large, and some tickets
>>>>>>>>> build
>>>>>>>>>>> off
>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> others.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I'm proposing
that we commit this work into a
>>>>>>>>> samza-sql
>>>>>>>>>>>>>>> submodule
>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>> master,
and treat this module as experimental. I
>>>>>>>> want
>>>>>>>>>>> to
>>>>>>>>>>>>> avoid
>>>>>>>>>>>>>>>>>>> branches,
>>>>>>>>>>>>>>>>>>>>> and I
also want to avoid revision control over
>>>>>>>> JIRA.
>>>>>>>>>>> This
>>>>>>>>>>>>>> means
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>>>> will
probably be a fair amount of commits/JIRAs
>>>>>>> on
>>>>>>>>> this
>>>>>>>>>>>>> module
>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>> iterate,
but I think that's OK.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Does
this sound good to everyone?
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>> Chris
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Milinda Pathirage
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> PhD Student | Research
Assistant
>>>>>>>>>>>>>>>>>> School of Informatics
and Computing | Data to Insight
>>>>>>>>> Center
>>>>>>>>>>>>>>>>>> Indiana University
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> twitter: milindalakmal
>>>>>>>>>>>>>>>>>> skype: milinda.pathirage
>>>>>>>>>>>>>>>>>> blog: http://milinda.pathirage.org
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> Milinda Pathirage
>>>>>>>>>> 
>>>>>>>>>> PhD Student | Research Assistant
>>>>>>>>>> School of Informatics and Computing | Data to Insight
Center
>>>>>>>>>> Indiana University
>>>>>>>>>> 
>>>>>>>>>> twitter: milindalakmal
>>>>>>>>>> skype: milinda.pathirage
>>>>>>>>>> blog: http://milinda.pathirage.org
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> Milinda Pathirage
>>>>>>>>> 
>>>>>>>>> PhD Student | Research Assistant
>>>>>>>>> School of Informatics and Computing | Data to Insight
Center
>>>>>>>>> Indiana University
>>>>>>>>> 
>>>>>>>>> twitter: milindalakmal
>>>>>>>>> skype: milinda.pathirage
>>>>>>>>> blog: http://milinda.pathirage.org
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> Milinda Pathirage
>>>>>>> 
>>>>>>> PhD Student | Research Assistant
>>>>>>> School of Informatics and Computing | Data to Insight Center
>>>>>>> Indiana University
>>>>>>> 
>>>>>>> twitter: milindalakmal
>>>>>>> skype: milinda.pathirage
>>>>>>> blog: http://milinda.pathirage.org
>>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>> 
>> 
>> 
>> 
>> --
>> Milinda Pathirage
>> 
>> PhD Student | Research Assistant
>> School of Informatics and Computing | Data to Insight Center
>> Indiana University
>> 
>> twitter: milindalakmal
>> skype: milinda.pathirage
>> blog: http://milinda.pathirage.org


Mime
View raw message