drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Altekruse <altekruseja...@gmail.com>
Subject Re: Time for a 1.5 release?
Date Thu, 28 Jan 2016 23:42:26 GMT
We could revert the change to the StoragePlugin interface to add the
getLogicalOptimizerRules
and getPhysicalOptimizerRules, and only put those in AbstractStoragePlugin
with default implementations.

We could then update the docs to tell users to not implement the interface
directly, and instead extend the abstract class, which I believe would
allow us to make changes like this in the future without breaking plugins.
At a major release version we could then just pull up all of the new
methods we added to the abstract class into the interface, or just get rid
of it all-together.

On Thu, Jan 28, 2016 at 2:59 PM, Venki Korukanti <venki.korukanti@gmail.com>
wrote:

> Here is the partial fix:
>
> https://github.com/vkorukanti/drill/commit/6646b0d08df274a22dff870ad5e6f6914b10fa0b
>
> If the existing StragePlugins are implementing AbstractStoragePlugin class
> this fix should resolve the backward compat issue. If they are implementing
> StoragePlugin interface directly then it won't resolve the issue. Not sure
> how to handle it in that case. Let me know if there any ways to resolve
> this issue completely.
>
> Thanks
> Venki
>
> On Thu, Jan 28, 2016 at 2:44 PM, Aman Sinha <asinha@maprtech.com> wrote:
>
> > I was able to do the build with maven 3.3.9 (after doing 'brew install
> > maven').  Here are my env variables related to maven:
> > M2_HOME=/usr/local/Cellar/maven/3.3.9/libexec
> > M2=/usr/local/Cellar/maven/3.3.9/libexec/bin
> > MAVEN_OPTS=-Xmx2048m -XX:MaxPermSize=256m
> >
> >
> > On Thu, Jan 28, 2016 at 2:20 PM, Jacques Nadeau <jacques@dremio.com>
> > wrote:
> >
> > > Aman, for reference on the Maven version issue. You can see the output
> of
> > > the shade plugin between the two Apache Builds. If you search for
> > > maven-shade-plugin you'll see that build 647 (maven 3.0.5) fails
> because
> > it
> > > includes a bunch of hadoop (and other libraries that are listed as
> being
> > > excluded). If you look at build 648 (maven 3.3.3), those items are
> > included
> > > so that the build completes successfully.
> > >
> > > https://builds.apache.org/job/drill-scm/647/consoleText
> > > https://builds.apache.org/job/drill-scm/648/consoleText
> > >
> > > --
> > > Jacques Nadeau
> > > CTO and Co-Founder, Dremio
> > >
> > > On Thu, Jan 28, 2016 at 2:01 PM, rahul challapalli <
> > > challapallirahul@gmail.com> wrote:
> > >
> > > > Jason,
> > > >
> > > > As of now I haven't narrowed down the exact commit which caused
> > > DRILL-4323
> > > > and I do not know the scope of the fix.
> > > >
> > > > - Rahul
> > > >
> > > > On Thu, Jan 28, 2016 at 1:39 PM, Jason Altekruse <
> > > altekrusejason@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > I think it makes sense to wait for Venki's fix of the API, as
> > > releasing a
> > > > > version with the change would make reverting/refactoring the API
> > > breaking
> > > > > change less useful.
> > > > >
> > > > > Do you have any idea about the scope of fixing the hive native
> > reader,
> > > or
> > > > > which change since the 1.4 release broke it? Is someone available
> to
> > > look
> > > > > at it today?
> > > > >
> > > > > On Thu, Jan 28, 2016 at 1:33 PM, rahul challapalli <
> > > > > challapallirahul@gmail.com> wrote:
> > > > >
> > > > > > Just found an issue with hive native parquet reader (DRILL-4323).
> > > This
> > > > > is a
> > > > > > regression from 1.4 and in my opinion should be treated as a
> > blocker.
> > > > > >
> > > > > > - Rahul
> > > > > >
> > > > > > On Thu, Jan 28, 2016 at 1:12 PM, Venki Korukanti <
> > > > > > venki.korukanti@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Sorry I didn't realize I was breaking the public interface
API.
> > Let
> > > > me
> > > > > > work
> > > > > > > on a patch to make it backward compatible.
> > > > > > >
> > > > > > > Thanks
> > > > > > > Venki
> > > > > > >
> > > > > > > On Thu, Jan 28, 2016 at 12:54 PM, Steven Phillips <
> > > steven@dremio.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I just wanted to bring up an issue that I just now
> discovered,
> > > that
> > > > > has
> > > > > > > > caused me a fair amount of grief.
> > > > > > > >
> > > > > > > > https://github.com/apache/drill/pull/300/commits
> > > > > > > >
> > > > > > > > DRILL-4198 changes a user-facing API, and causes
> StoragePlugins
> > > > that
> > > > > > were
> > > > > > > > compiled against currently released versions of Drill
to no
> > > longer
> > > > > > > > functional properly. I would prefer that this breaking
change
> > be
> > > > > > modified
> > > > > > > > to be backward compatible if possible.
> > > > > > > >
> > > > > > > > On Thu, Jan 28, 2016 at 11:23 AM, Jason Altekruse
<
> > > > > > > > altekrusejason@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Aman,
> > > > > > > > >
> > > > > > > > > This is the failure that he was seeing. He figured
out that
> > the
> > > > new
> > > > > > > > > exclusions in jdbc-all were not being respected
when the
> > build
> > > > was
> > > > > > run
> > > > > > > > with
> > > > > > > > > an older Maven version, causing the jar size
to increase
> > > > > > significantly.
> > > > > > > > He
> > > > > > > > > added an enforcer to make sure the JAR didn't
grow
> > > unexpectedly.
> > > > > Can
> > > > > > > you
> > > > > > > > > try to update your maven version and re-run the
build?
> > > > > > > > >
> > > > > > > > > - Jason
> > > > > > > > >
> > > > > > > > > On Thu, Jan 28, 2016 at 11:18 AM, Aman Sinha
<
> > > > asinha@maprtech.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Jacques, I am getting the following build
failure on the
> > > latest
> > > > > > > master
> > > > > > > > > > branch...is this what you saw for the Apache
build ?  My
> > mvn
> > > > > > version
> > > > > > > > > output
> > > > > > > > > > is shown below.  Should we all be upgrading
to a newer
> mvn
> > ?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > [INFO] --- maven-enforcer-plugin:1.3.1:enforce
> > > > > > > > > > (enforce-jdbc-jar-compactness) @ drill-jdbc-all
---
> > > > > > > > > > [WARNING] Rule 0:
> > > > > > org.apache.maven.plugins.enforcer.RequireFilesSize
> > > > > > > > > failed
> > > > > > > > > > with message:
> > > > > > > > > > The file drill-jdbc-all-1.5.0-SNAPSHOT.jar
is outside the
> > > > > expected
> > > > > > > size
> > > > > > > > > > range.
> > > > > > > > > >
> > > > > > > > > >                   This is likely due to
you adding new
> > > > > dependencies
> > > > > > > to
> > > > > > > > a
> > > > > > > > > > java-exec and not updating the excludes
in this module.
> > This
> > > is
> > > > > > > > important
> > > > > > > > > > as it minimizes the size of the dependency
of Drill
> > > application
> > > > > > > users.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /Users/asinha/incubator-drill/exec/jdbc-all/target/drill-jdbc-all-1.5.0-SNAPSHOT.jar
> > > > > > > > > > size (44664121) too large. Max. is
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> 20000000/Users/asinha/incubator-drill/exec/jdbc-all/target/drill-jdbc-all-1.5.0-SNAPSHOT.jar
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Administrators-MacBook-Pro-144:incubator-drill
asinha$
> mvn
> > > > > > --version
> > > > > > > > > > Apache Maven 3.0.5
> > > (r01de14724cdef164cd33c7c8c2fe155faf9602da;
> > > > > > > > 2013-02-19
> > > > > > > > > > 05:51:28-0800)
> > > > > > > > > > Maven home: /opt/local/share/java/maven3
> > > > > > > > > > Java version: 1.7.0_45, vendor: Oracle Corporation
> > > > > > > > > > Java home:
> > > > > > > > > >
> > > > > /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/jre
> > > > > > > > > > Default locale: en_US, platform encoding:
UTF-8
> > > > > > > > > > OS name: "mac os x", version: "10.9.5",
arch: "x86_64",
> > > family:
> > > > > > "mac"
> > > > > > > > > >
> > > > > > > > > > On Thu, Jan 28, 2016 at 8:20 AM, Jacques
Nadeau <
> > > > > > jacques@dremio.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Build back to normal. It looks like
the Apache server
> was
> > > > using
> > > > > > an
> > > > > > > > old
> > > > > > > > > > > version of Maven. Once I switched to
something more
> > recent,
> > > > the
> > > > > > > build
> > > > > > > > > > > passed.
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Jacques Nadeau
> > > > > > > > > > > CTO and Co-Founder, Dremio
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Jan 28, 2016 at 7:02 AM, Jacques
Nadeau <
> > > > > > > jacques@dremio.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hmm... this merge caused the Apache
build to fail.
> > > > > > > Investigating...
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > Jacques Nadeau
> > > > > > > > > > > > CTO and Co-Founder, Dremio
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Jan 28, 2016 at 6:31 AM,
Jacques Nadeau <
> > > > > > > > jacques@dremio.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> I got clean regression runs
as well. I've merged the
> > > > patch.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Jason, you want to start the
release process?
> > > > > > > > > > > >>
> > > > > > > > > > > >> --
> > > > > > > > > > > >> Jacques Nadeau
> > > > > > > > > > > >> CTO and Co-Founder, Dremio
> > > > > > > > > > > >>
> > > > > > > > > > > >> On Wed, Jan 27, 2016 at 10:42
PM, Abhishek Girish <
> > > > > > > > agirish@mapr.com
> > > > > > > > > >
> > > > > > > > > > > >> wrote:
> > > > > > > > > > > >>
> > > > > > > > > > > >>> Had two clean Functional
runs. TPC-H SF100 was also
> > > > > > successful.
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> On Wed, Jan 27, 2016 at
10:07 PM, rahul
> challapalli <
> > > > > > > > > > > >>> challapallirahul@gmail.com>
wrote:
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> > Kicked off a functional
run with your branch.
> Will
> > > let
> > > > > you
> > > > > > > know
> > > > > > > > > > once
> > > > > > > > > > > it
> > > > > > > > > > > >>> > finishes
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > - Rahul
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > On Wed, Jan 27, 2016
at 9:56 PM, Jacques Nadeau <
> > > > > > > > > > jacques@dremio.com>
> > > > > > > > > > > >>> > wrote:
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > > 4196 was merged
today. I have an updated patch
> > for
> > > > 4291
> > > > > > > that
> > > > > > > > is
> > > > > > > > > > > >>> ready.
> > > > > > > > > > > >>> > > Unfortunately,
it seems that something isn't
> > > working
> > > > > with
> > > > > > > our
> > > > > > > > > > > >>> extended
> > > > > > > > > > > >>> > > tests so I haven't
been able to run an extended
> > > > > > regression.
> > > > > > > > > Unit
> > > > > > > > > > > >>> tests
> > > > > > > > > > > >>> > > pass. Is someone
else possibly able to run a
> > > > regression
> > > > > > > suite
> > > > > > > > > > > against
> > > > > > > > > > > >>> > this
> > > > > > > > > > > >>> > > branch [1] so
we can confirm things look good
> and
> > > > start
> > > > > > the
> > > > > > > > > > release
> > > > > > > > > > > >>> > > process?
> > > > > > > > > > > >>> > >
> > > > > > > > > > > >>> > > thanks,
> > > > > > > > > > > >>> > > Jacques
> > > > > > > > > > > >>> > >
> > > > > > > > > > > >>> > > [1]
> > > > > https://github.com/jacques-n/drill/tree/DRILL-4291v2
> > > > > > > > > > > >>> > >
> > > > > > > > > > > >>> > > --
> > > > > > > > > > > >>> > > Jacques Nadeau
> > > > > > > > > > > >>> > > CTO and Co-Founder,
Dremio
> > > > > > > > > > > >>> > >
> > > > > > > > > > > >>> > > On Mon, Jan
25, 2016 at 11:20 AM, Jacques
> Nadeau
> > <
> > > > > > > > > > > jacques@dremio.com
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > > wrote:
> > > > > > > > > > > >>> > >
> > > > > > > > > > > >>> > > > I think
the main things are 4196 and 4291
> > should
> > > be
> > > > > > > > > completed.
> > > > > > > > > > I
> > > > > > > > > > > >>> know
> > > > > > > > > > > >>> > > Amit
> > > > > > > > > > > >>> > > > was able
to reproduce 4196 locally this
> weekend
> > > so
> > > > I
> > > > > > > think
> > > > > > > > > > we're
> > > > > > > > > > > >>> close
> > > > > > > > > > > >>> > on
> > > > > > > > > > > >>> > > > that.
> > > > > > > > > > > >>> > > >
> > > > > > > > > > > >>> > > > --
> > > > > > > > > > > >>> > > > Jacques
Nadeau
> > > > > > > > > > > >>> > > > CTO and
Co-Founder, Dremio
> > > > > > > > > > > >>> > > >
> > > > > > > > > > > >>> > > > On Mon,
Jan 25, 2016 at 10:14 AM, Zelaine
> Fong
> > <
> > > > > > > > > > > zfong@maprtech.com
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > > wrote:
> > > > > > > > > > > >>> > > >
> > > > > > > > > > > >>> > > >> Any
updates on this?  What's blocking us
> from
> > > > taking
> > > > > > > this
> > > > > > > > > to a
> > > > > > > > > > > >>> vote --
> > > > > > > > > > > >>> > > the
> > > > > > > > > > > >>> > > >> sort
merge join issues?
> > > > > > > > > > > >>> > > >>
> > > > > > > > > > > >>> > > >> --
Zelaine
> > > > > > > > > > > >>> > > >>
> > > > > > > > > > > >>> > > >> On
Tue, Jan 19, 2016 at 4:18 PM, Jacques
> > Nadeau
> > > <
> > > > > > > > > > > >>> jacques@dremio.com>
> > > > > > > > > > > >>> > > >> wrote:
> > > > > > > > > > > >>> > > >>
> > > > > > > > > > > >>> > > >> >
Bumping this thread...
> > > > > > > > > > > >>> > > >> >
> > > > > > > > > > > >>> > > >> >
Here are the issues that were mentioned in
> > > this
> > > > > > thread
> > > > > > > > > along
> > > > > > > > > > > >>> with a
> > > > > > > > > > > >>> > > >> >
proposed categorization:
> > > > > > > > > > > >>> > > >> >
> > > > > > > > > > > >>> > > >> >
Release Blockers
> > > > > > > > > > > >>> > > >> >
In-progress Amit
> > > > > > > > > > > >>> https://issues.apache.org/jira/browse/DRILL-4190
> > > > > > > > > > > >>> > > >> >
In-progress Amit
> > > > > > > > > > > >>> https://issues.apache.org/jira/browse/DRILL-4196
> > > > > > > > > > > >>> > > >> >
Ready to merge Jacques
> > > > > > > > > > > >>> > >
> https://issues.apache.org/jira/browse/DRILL-4246
> > > > > > > > > > > >>> > > >> >
In-review Jinfeng
> > > > > > > > > > > >>> https://issues.apache.org/jira/browse/DRILL-4256
> > > > > > > > > > > >>> > > >> >
In-progress Jacques
> > > > > > > > > > > >>> > https://issues.apache.org/jira/browse/DRILL-4278
> > > > > > > > > > > >>> > > >> >
Ready to merge Laurent
> > > > > > > > > > > >>> > >
> https://issues.apache.org/jira/browse/DRILL-4285
> > > > > > > > > > > >>> > > >> >
Nice to Have
> > > > > > > > > > > >>> > > >> >
Open Jason/Hakim
> > > > > > > > > > > >>> https://issues.apache.org/jira/browse/DRILL-4247
> > > > > > > > > > > >>> > > >> >
In-progress Jason
> > > > > > > > > > > >>> https://issues.apache.org/jira/browse/DRILL-4203
> > > > > > > > > > > >>> > > >> >
Open Jacques
> > > > > > > > > > https://issues.apache.org/jira/browse/DRILL-4266
> > > > > > > > > > > >>> > > >> >
Ready to merge Jacques
> > > > > > > > > > > >>> > >
> https://issues.apache.org/jira/browse/DRILL-4131
> > > > > > > > > > > >>> > > >> >
> > > > > > > > > > > >>> > > >> >
What do others think? Let's try to get the
> > > > > blockers
> > > > > > > > > wrapped
> > > > > > > > > > up
> > > > > > > > > > > >>> in
> > > > > > > > > > > >>> > the
> > > > > > > > > > > >>> > > >> next
> > > > > > > > > > > >>> > > >> >
day or two and start a release vote...
> > > > > > > > > > > >>> > > >> >
> > > > > > > > > > > >>> > > >> >
> > > > > > > > > > > >>> > > >> >
> > > > > > > > > > > >>> > > >> >
--
> > > > > > > > > > > >>> > > >> >
Jacques Nadeau
> > > > > > > > > > > >>> > > >> >
CTO and Co-Founder, Dremio
> > > > > > > > > > > >>> > > >> >
> > > > > > > > > > > >>> > > >> >
On Mon, Jan 4, 2016 at 1:48 PM, Jason
> > > Altekruse
> > > > <
> > > > > > > > > > > >>> > > >> altekrusejason@gmail.com>
> > > > > > > > > > > >>> > > >> >
wrote:
> > > > > > > > > > > >>> > > >> >
> > > > > > > > > > > >>> > > >> >
> Hello All,
> > > > > > > > > > > >>> > > >> >
>
> > > > > > > > > > > >>> > > >> >
> With the allocator changes merged and
> > about
> > > a
> > > > > > month
> > > > > > > > > since
> > > > > > > > > > > the
> > > > > > > > > > > >>> last
> > > > > > > > > > > >>> > > >> >
release
> > > > > > > > > > > >>> > > >> >
> I think it would be good to start a vote
> > > > soon. I
> > > > > > > would
> > > > > > > > > > like
> > > > > > > > > > > to
> > > > > > > > > > > >>> > > >> volunteer
> > > > > > > > > > > >>> > > >> >
to
> > > > > > > > > > > >>> > > >> >
> be release manager.
> > > > > > > > > > > >>> > > >> >
>
> > > > > > > > > > > >>> > > >> >
> I know that there were some issues that
> > were
> > > > > > > > identified
> > > > > > > > > > > after
> > > > > > > > > > > >>> the
> > > > > > > > > > > >>> > > >> >
transfer
> > > > > > > > > > > >>> > > >> >
> patch was merged. I think that these
> > issues
> > > > > should
> > > > > > > be
> > > > > > > > > > fixed
> > > > > > > > > > > >>> before
> > > > > > > > > > > >>> > > we
> > > > > > > > > > > >>> > > >> >
cut a
> > > > > > > > > > > >>> > > >> >
> release candidate.
> > > > > > > > > > > >>> > > >> >
>
> > > > > > > > > > > >>> > > >> >
> From looking at the associated JIRAs it
> > > looked
> > > > > > like
> > > > > > > > > there
> > > > > > > > > > > was
> > > > > > > > > > > >>> a
> > > > > > > > > > > >>> > > >> possible
> > > > > > > > > > > >>> > > >> >
> short term fix just adjusting the
> > > > > > > > > > max_query_memory_per_node
> > > > > > > > > > > >>> > option,
> > > > > > > > > > > >>> > > >> and
> > > > > > > > > > > >>> > > >> >
> some more involved work to change how we
> > > > > determine
> > > > > > > the
> > > > > > > > > > > correct
> > > > > > > > > > > >>> > time
> > > > > > > > > > > >>> > > to
> > > > > > > > > > > >>> > > >> >
> spill during external sort. I believe it
> > > makes
> > > > > > sense
> > > > > > > > to
> > > > > > > > > > make
> > > > > > > > > > > >>> > > external
> > > > > > > > > > > >>> > > >> >
sort
> > > > > > > > > > > >>> > > >> >
> work well with the newly improved memory
> > > > > > accounting
> > > > > > > > > before
> > > > > > > > > > > >>> > cutting a
> > > > > > > > > > > >>> > > >> >
> release, but I'm not sure how much work
> is
> > > > left
> > > > > to
> > > > > > > be
> > > > > > > > > done
> > > > > > > > > > > >>> there.
> > > > > > > > > > > >>> > > [1]
> > > > > > > > > > > >>> > > >> >
>
> > > > > > > > > > > >>> > > >> >
> Please respond with your thoughts on a
> > > release
> > > > > > soon
> > > > > > > > and
> > > > > > > > > > any
> > > > > > > > > > > >>> JIRAs
> > > > > > > > > > > >>> > > you
> > > > > > > > > > > >>> > > >> >
would
> > > > > > > > > > > >>> > > >> >
> like to include in the release.
> > > > > > > > > > > >>> > > >> >
>
> > > > > > > > > > > >>> > > >> >
> [1] -
> > > > > > > > https://issues.apache.org/jira/browse/DRILL-4243
> > > > > > > > > > > >>> > > >> >
>
> > > > > > > > > > > >>> > > >> >
> Thanks,
> > > > > > > > > > > >>> > > >> >
> Jason
> > > > > > > > > > > >>> > > >> >
>
> > > > > > > > > > > >>> > > >> >
> > > > > > > > > > > >>> > > >>
> > > > > > > > > > > >>> > > >
> > > > > > > > > > > >>> > > >
> > > > > > > > > > > >>> > >
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>>
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message