samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Riccomini <criccom...@apache.org>
Subject Re: [DISCUSS] SQL workflow
Date Thu, 12 Feb 2015 18:30:08 GMT
@Jakob, totally agree. I think the semantics of the SQL module won't
change. RTC, experimental, more lax on tests/docs. When we commit into
master, we can enforce much stronger test/doc requirements. That's what I'm
thinking.

On Thu, Feb 12, 2015 at 9:26 AM, Jakob Homan <jghoman@gmail.com> wrote:

> +1, assuming we're keeping RTC on the branch (, Jakob said pedantically).
>
> On 12 February 2015 at 08:25, Milinda Pathirage <mpathira@umail.iu.edu>
> wrote:
> > I am +1 for this approach.
> >
> > Milinda
> >
> > On Thu, Feb 12, 2015 at 11:22 AM, Chris Riccomini <criccomini@apache.org
> >
> > wrote:
> >
> >> Hey Julian,
> >>
> >> > The rule of thumb I’ve learned integrating Calcite into Hive is that
> only
> >> a branch should depend on snapshot versions of another component
> >>
> >> This is very practical. I hadn't considered the Calcite dependency issue
> >> when I pushed for merging into master rather than a branch.
> >>
> >> Based on this, it seems like the best thing to do is:
> >>
> >> 1) Work off of Calcite SNAPSHOT Maven publication.
> >> 2) Create a samza-sql branch.
> >> 3) Migrate all existing samza-sql commits from master into the samza-sql
> >> branch.
> >>
> >> Sorry for the churn on this all. I hadn't considered the binary
> dependency
> >> issue. Is everyone OK with this?
> >>
> >> Cheers,
> >> Chris
> >>
> >> On Wed, Feb 11, 2015 at 10:51 AM, Julian Hyde <julian@hydromatic.net>
> >> wrote:
> >>
> >> > This seems more like a branch than a classifier. Calcite is
> developing in
> >> > a branch, and would produce snapshots from that branch. The rule of
> thumb
> >> > I’ve learned integrating Calcite into Hive is that only a branch
> should
> >> > depend on snapshot versions of another component. (Hive broke this
> rule
> >> and
> >> > got into a sticky situation where their trunk depended on a Calcite
> >> > snapshot and they couldn’t release because Calcite was stuck in
> >> pre-release
> >> > vote limbo.)
> >> >
> >> > I’m going to try to produce maven artifacts with {groupId:
> >> > “org.apache.calcite”, version: "1.1.0-stream-incubating-SNAPSHOT”,
> >> > artifactId: “calcite-core”, “calcite-avatica”, etc.} and push them
to
> >> > Apache nexus. I do recommend that you use it on a branch.
> >> >
> >> > However, I don’t think there would be a problem getting the
> functionality
> >> > into calcite’s master branch in time for Calcite’s next release, in
> >> about 4
> >> > weeks. (We commit to releasing approximately monthly.) The
> functionality
> >> > and APIs would be marked “experimental, subject to change” but at
> least
> >> you
> >> > would be able to merge your work to master at that point. Doubtless
> you
> >> > would need new features & bug fixes in Calcite after that, and so work
> >> that
> >> > depended on that would have to stay in a branch until Calcite was
> >> released.
> >> >
> >> > Julian
> >> >
> >> > On Feb 11, 2015, at 10:34 AM, Felix GV <fvillegas@linkedin.com.INVALID
> >
> >> > wrote:
> >> >
> >> > > I guess one concern with Calcite SNAPSHOT releases is that they
> imply
> >> > they'll be included in the next release, which may not be the case if
> >> > they're still considered very experimental.
> >> > >
> >> > > Alternatively, perhaps Julian can create a new version name with a
> >> > classifier for streaming.
> >> > >
> >> > >
> >> >
> >>
> http://maven.apache.org/plugins/maven-deploy-plugin/examples/deploying-with-classifiers.html
> >> > >
> >> > > Then he could release SNAPSHOT artifacts for both the main/master
> code
> >> > base as well as the streaming branch, no matter how experimental it
> is...
> >> > >
> >> > > --
> >> > >
> >> > > Felix GV
> >> > > Data Infrastructure Engineer
> >> > > Distributed Data Systems
> >> > > LinkedIn
> >> > >
> >> > > fgv@linkedin.com
> >> > > linkedin.com/in/felixgv
> >> > >
> >> > > ________________________________________
> >> > > From: Chris Riccomini [criccomini@apache.org]
> >> > > Sent: Wednesday, February 11, 2015 10:23 AM
> >> > > To: Chris Riccomini
> >> > > Cc: dev@samza.apache.org
> >> > > Subject: Re: [DISCUSS] SQL workflow
> >> > >
> >> > > Hey Julian,
> >> > >
> >> > > It looks like Calcite is at least setup to publish to Maven:
> >> > >
> >> > >  https://repository.apache.org/#nexus-search;quick~calcite
> >> > >
> >> > > Julian, can you publish SNAPSHOT releases with your streaming
> changes
> >> to
> >> > it?
> >> > >
> >> > > Thanks!
> >> > > Chris
> >> > >
> >> > > On Wed, Feb 11, 2015 at 10:08 AM, Chris Riccomini <
> >> criccomini@apache.org
> >> > >
> >> > > wrote:
> >> > >
> >> > >> Hey Milinda,
> >> > >>
> >> > >> I've just committed SAMZA-484, so you should be able to re-base
and
> >> get
> >> > >> all the latest code.
> >> > >>
> >> > >>> But we need to add Calcite as a dependency and Calcite streaming
> >> > >> support is only in Julian's branch
> >> > >>
> >> > >> We'll have to have Julian publish a release to Apache. This can
be
> a
> >> > >> SNAPSHOT to Apache's SNAPSHOT repo, or a real Calcite release.
We
> >> can't
> >> > >> check in binaries to the repo (licensing, release complexity,
and
> just
> >> > >> generally a bad idea). We'll have to coordinate with him.
> >> > >>
> >> > >> Cheers,
> >> > >> Chris
> >> > >>
> >> > >> On Wed, Feb 11, 2015 at 9:43 AM, Milinda Pathirage <
> >> > mpathira@umail.iu.edu>
> >> > >> wrote:
> >> > >>
> >> > >>> No, SAMZA-483 doesn't depend on SAMZA-484. But we need to
add
> Calcite
> >> > as a
> >> > >>> dependency and Calcite streaming support is only in Julian's
> branch (
> >> > >>> https://github.com/julianhyde/incubator-calcite/tree/chi).
What
> can
> >> > we do
> >> > >>> about that?
> >> > >>>
> >> > >>> Milinda
> >> > >>>
> >> > >>> On Wed, Feb 11, 2015 at 12:30 PM, Chris Riccomini <
> >> > criccomini@apache.org>
> >> > >>> wrote:
> >> > >>>
> >> > >>>> Hey Milinda,
> >> > >>>>
> >> > >>>> That'd be great. Do you have any dependency on SAMZA-484?
I
> haven't
> >> > had
> >> > >>> a
> >> > >>>> chance to commit that one yet.
> >> > >>>>
> >> > >>>> Cheers,
> >> > >>>> Chris
> >> > >>>>
> >> > >>>> On Wed, Feb 11, 2015 at 9:23 AM, Milinda Pathirage <
> >> > >>> mpathira@umail.iu.edu>
> >> > >>>> wrote:
> >> > >>>>
> >> > >>>>> /need/want
> >> > >>>>>
> >> > >>>>> Milinda
> >> > >>>>>
> >> > >>>>> On Wed, Feb 11, 2015 at 12:22 PM, Milinda Pathirage
<
> >> > >>>> mpathira@umail.iu.edu
> >> > >>>>>>
> >> > >>>>> wrote:
> >> > >>>>>
> >> > >>>>>> Hi Chris,
> >> > >>>>>>
> >> > >>>>>> Do you need me to create the SAMZA-483 patch against
latest
> master
> >> > >>> with
> >> > >>>>>> SAMZA-482 patch? I think that will make it easier
to review the
> >> > >>> patch?
> >> > >>>>>>
> >> > >>>>>> Thanks
> >> > >>>>>> Milinda
> >> > >>>>>>
> >> > >>>>>> On Mon, Feb 9, 2015 at 11:39 AM, Chris Riccomini
<
> >> > >>>> criccomini@apache.org>
> >> > >>>>>> wrote:
> >> > >>>>>>
> >> > >>>>>>> Cool. Looks like we've got consensus. I'll
move forward with
> RTC
> >> on
> >> > >>>> some
> >> > >>>>>>> of
> >> > >>>>>>> the early SAMZA-390 tickets. (SAMZA-482, SAMZA-483,
SAMZA-484)
> >> > >>>>>>>
> >> > >>>>>>> On Mon, Feb 9, 2015 at 8:35 AM, Yan Fang <
> yanfang724@gmail.com>
> >> > >>>> wrote:
> >> > >>>>>>>
> >> > >>>>>>>> +1 on this.
> >> > >>>>>>>>
> >> > >>>>>>>> Fang, Yan
> >> > >>>>>>>> yanfang724@gmail.com
> >> > >>>>>>>> +1 (206) 849-4108
> >> > >>>>>>>>
> >> > >>>>>>>> On Fri, Feb 6, 2015 at 4:38 PM, Chris
Riccomini <
> >> > >>>>> criccomini@apache.org>
> >> > >>>>>>>> wrote:
> >> > >>>>>>>>
> >> > >>>>>>>>> Hey all,
> >> > >>>>>>>>>
> >> > >>>>>>>>> Are we +1 on this? I think Jakob was
the only one who was
> >> > >>> curious
> >> > >>>>>>> about
> >> > >>>>>>>> it.
> >> > >>>>>>>>>
> >> > >>>>>>>>> Cheers,
> >> > >>>>>>>>> Chris
> >> > >>>>>>>>>
> >> > >>>>>>>>> On Thu, Feb 5, 2015 at 1:22 PM, Yi
Pan <nickpan47@gmail.com
> >
> >> > >>>> wrote:
> >> > >>>>>>>>>
> >> > >>>>>>>>>> Hi, Jakob,
> >> > >>>>>>>>>>
> >> > >>>>>>>>>>
> >> > >>>>>>>>>>>> Eh? Not sure what this
means...
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>> I mean SAMZA-484 depends on
SAMZA-482, and neither are
> >> > >>>>> committed.
> >> > >>>>>>> So
> >> > >>>>>>>>>> Navina
> >> > >>>>>>>>>>> is having to post Yi's patch,
as well as her own, on the
> >> > >>> JIRA.
> >> > >>>>> It
> >> > >>>>>>>> makes
> >> > >>>>>>>>>> it
> >> > >>>>>>>>>>> really hard to do code reviews
because you can't tell
> >> > >>> whether
> >> > >>>> Yi
> >> > >>>>>>> made
> >> > >>>>>>>>> the
> >> > >>>>>>>>>>> changes or Navina did.
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>> Just to add to the point. It is
also difficult to always
> see
> >> > >>> a
> >> > >>>>> long
> >> > >>>>>>>> list
> >> > >>>>>>>>> of
> >> > >>>>>>>>>> changed files if the RB request
is always based on the
> >> > >>> master.
> >> > >>>> It
> >> > >>>>> is
> >> > >>>>>>>>>> possible to have RB request based
on another RB request (I
> >> > >>> have
> >> > >>>>>>> tried
> >> > >>>>>>>> it
> >> > >>>>>>>>>> before). But what happens if the
base RB request is
> >> > >>>>>>>> cancelled/discarded?
> >> > >>>>>>>>> RB
> >> > >>>>>>>>>> is not designed to track the revision
changes in a
> dependency
> >> > >>>>> chain.
> >> > >>>>>>>>>>
> >> > >>>>>>>>>>
> >> > >>>>>>>>>>
> >> > >>>>>>>>>>> Cheers,
> >> > >>>>>>>>>>> Chris
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>> On Thu, Feb 5, 2015 at 12:16
PM, Jakob Homan <
> >> > >>>> jghoman@gmail.com
> >> > >>>>>>
> >> > >>>>>>>>> wrote:
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>>> I want to avoid branches,
> >> > >>>>>>>>>>>> Just curious, any reason
for this?
> >> > >>>>>>>>>>>>
> >> > >>>>>>>>>>>>> and I also want to
avoid revision control over JIRA
> >> > >>>>>>>>>>>> Eh? Not sure what this
means...
> >> > >>>>>>>>>>>>
> >> > >>>>>>>>>>>> Thanks,
> >> > >>>>>>>>>>>> jg
> >> > >>>>>>>>>>>>
> >> > >>>>>>>>>>>> On 4 February 2015 at
17:11, Chris Riccomini <
> >> > >>>>>>>> criccomini@apache.org>
> >> > >>>>>>>>>>>> wrote:
> >> > >>>>>>>>>>>>> Hey all,
> >> > >>>>>>>>>>>>>
> >> > >>>>>>>>>>>>> @Jakob, yeah I was
thinking we'll follow our normal
> >> > >>> flow.
> >> > >>>>>>> RTC. I
> >> > >>>>>>>>> just
> >> > >>>>>>>>>>>>> wanted to set expectation
that the code committed
> >> > >>> might be
> >> > >>>>>>> not up
> >> > >>>>>>>>> to
> >> > >>>>>>>>>>> our
> >> > >>>>>>>>>>>>> normal quality initially
(missing docs, no tests, etc).
> >> > >>>>> Until
> >> > >>>>>>> the
> >> > >>>>>>>>>>> quality
> >> > >>>>>>>>>>>>> is raised, we should
think of this module as
> >> > >>> experimental.
> >> > >>>>>>>>>>>>>
> >> > >>>>>>>>>>>>> @Milinda, awesome!
Thanks. :)
> >> > >>>>>>>>>>>>>
> >> > >>>>>>>>>>>>> Cheers,
> >> > >>>>>>>>>>>>> Chris
> >> > >>>>>>>>>>>>>
> >> > >>>>>>>>>>>>> On Wed, Feb 4, 2015
at 11:57 AM, Milinda Pathirage <
> >> > >>>>>>>>>>>> mpathira@umail.iu.edu>
> >> > >>>>>>>>>>>>> wrote:
> >> > >>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>> Hi Chris,
> >> > >>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>> Hope we no longer
need the SQL API. I'll create a RB
> >> > >>> for
> >> > >>>>>>> Calcite
> >> > >>>>>>>>>>>>>> integration.
> >> > >>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>> Thanks
> >> > >>>>>>>>>>>>>> Milinda
> >> > >>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>> On Wed, Feb 4,
2015 at 1:31 PM, Chris Riccomini <
> >> > >>>>>>>>>>> criccomini@apache.org>
> >> > >>>>>>>>>>>>>> wrote:
> >> > >>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>> I think so.
There was some RB downtime, but it just
> >> > >>> got
> >> > >>>>>>> fixed.
> >> > >>>>>>>>> Yi,
> >> > >>>>>>>>>>>>>> Navina,
> >> > >>>>>>>>>>>>>>> Milinda, can
you make sure your JIRAs have up to
> >> > >>> date
> >> > >>>>> RBs?
> >> > >>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>> On Wed, Feb
4, 2015 at 10:24 AM, sriram <
> >> > >>>>>>> sriram.sub@gmail.com
> >> > >>>>>>>>>
> >> > >>>>>>>>>>> wrote:
> >> > >>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>> Can we
have updated RBs for all the three sub
> >> > >>> tasks
> >> > >>>>>>> before
> >> > >>>>>>>> we
> >> > >>>>>>>>>>>> commit?
> >> > >>>>>>>>>>>>>>> This
> >> > >>>>>>>>>>>>>>>> would
help us to review even after we commit.
> >> > >>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>> On Wed,
Feb 4, 2015 at 10:15 AM, Chris Riccomini <
> >> > >>>>>>>>>>>>>> criccomini@apache.org>
> >> > >>>>>>>>>>>>>>>> wrote:
> >> > >>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>> Hey
all,
> >> > >>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>> Yi,
Navina, and Milinda have been working on
> >> > >>>>> SAMZA-390
> >> > >>>>>>>>>>> sub-tickets
> >> > >>>>>>>>>>>>>>>> related
> >> > >>>>>>>>>>>>>>>>> to
SQL operators. We're getting to the point
> >> > >>> where
> >> > >>>>> the
> >> > >>>>>>>>> amount
> >> > >>>>>>>>>> of
> >> > >>>>>>>>>>>> work
> >> > >>>>>>>>>>>>>>>>> floating
around is quite large, and some tickets
> >> > >>>>> build
> >> > >>>>>>> off
> >> > >>>>>>>>> of
> >> > >>>>>>>>>>>> others.
> >> > >>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>> I'm
proposing that we commit this work into a
> >> > >>>>> samza-sql
> >> > >>>>>>>>>>> submodule
> >> > >>>>>>>>>>>> on
> >> > >>>>>>>>>>>>>>>>> master,
and treat this module as experimental. I
> >> > >>>> want
> >> > >>>>>>> to
> >> > >>>>>>>>> avoid
> >> > >>>>>>>>>>>>>>> branches,
> >> > >>>>>>>>>>>>>>>>> and
I also want to avoid revision control over
> >> > >>>> JIRA.
> >> > >>>>>>> This
> >> > >>>>>>>>>> means
> >> > >>>>>>>>>>>> that
> >> > >>>>>>>>>>>>>>>> there
> >> > >>>>>>>>>>>>>>>>> will
probably be a fair amount of commits/JIRAs
> >> > >>> on
> >> > >>>>> this
> >> > >>>>>>>>> module
> >> > >>>>>>>>>>> as
> >> > >>>>>>>>>>>> we
> >> > >>>>>>>>>>>>>>>>> iterate,
but I think that's OK.
> >> > >>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>> Does
this sound good to everyone?
> >> > >>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>> Cheers,
> >> > >>>>>>>>>>>>>>>>> Chris
> >> > >>>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>> --
> >> > >>>>>>>>>>>>>> Milinda Pathirage
> >> > >>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>> PhD Student |
Research Assistant
> >> > >>>>>>>>>>>>>> School of Informatics
and Computing | Data to Insight
> >> > >>>>> Center
> >> > >>>>>>>>>>>>>> Indiana University
> >> > >>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>>> twitter: milindalakmal
> >> > >>>>>>>>>>>>>> skype: milinda.pathirage
> >> > >>>>>>>>>>>>>> blog: http://milinda.pathirage.org
> >> > >>>>>>>>>>>>>>
> >> > >>>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>
> >> > >>>>>>>>>
> >> > >>>>>>>>
> >> > >>>>>>>
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>> --
> >> > >>>>>> Milinda Pathirage
> >> > >>>>>>
> >> > >>>>>> PhD Student | Research Assistant
> >> > >>>>>> School of Informatics and Computing | Data to
Insight Center
> >> > >>>>>> Indiana University
> >> > >>>>>>
> >> > >>>>>> twitter: milindalakmal
> >> > >>>>>> skype: milinda.pathirage
> >> > >>>>>> blog: http://milinda.pathirage.org
> >> > >>>>>>
> >> > >>>>>
> >> > >>>>>
> >> > >>>>>
> >> > >>>>> --
> >> > >>>>> Milinda Pathirage
> >> > >>>>>
> >> > >>>>> PhD Student | Research Assistant
> >> > >>>>> School of Informatics and Computing | Data to Insight
Center
> >> > >>>>> Indiana University
> >> > >>>>>
> >> > >>>>> twitter: milindalakmal
> >> > >>>>> skype: milinda.pathirage
> >> > >>>>> blog: http://milinda.pathirage.org
> >> > >>>>>
> >> > >>>>
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>> --
> >> > >>> Milinda Pathirage
> >> > >>>
> >> > >>> PhD Student | Research Assistant
> >> > >>> School of Informatics and Computing | Data to Insight Center
> >> > >>> Indiana University
> >> > >>>
> >> > >>> twitter: milindalakmal
> >> > >>> skype: milinda.pathirage
> >> > >>> blog: http://milinda.pathirage.org
> >> > >>>
> >> > >>
> >> > >>
> >> >
> >> >
> >>
> >
> >
> >
> > --
> > Milinda Pathirage
> >
> > PhD Student | Research Assistant
> > School of Informatics and Computing | Data to Insight Center
> > Indiana University
> >
> > twitter: milindalakmal
> > skype: milinda.pathirage
> > blog: http://milinda.pathirage.org
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message