drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacques Nadeau <jacq...@dremio.com>
Subject Re: Time for a 1.5 release?
Date Thu, 28 Jan 2016 22:20:22 GMT
Aman, for reference on the Maven version issue. You can see the output of
the shade plugin between the two Apache Builds. If you search for
maven-shade-plugin you'll see that build 647 (maven 3.0.5) fails because it
includes a bunch of hadoop (and other libraries that are listed as being
excluded). If you look at build 648 (maven 3.3.3), those items are included
so that the build completes successfully.

https://builds.apache.org/job/drill-scm/647/consoleText
https://builds.apache.org/job/drill-scm/648/consoleText

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Thu, Jan 28, 2016 at 2:01 PM, rahul challapalli <
challapallirahul@gmail.com> wrote:

> Jason,
>
> As of now I haven't narrowed down the exact commit which caused DRILL-4323
> and I do not know the scope of the fix.
>
> - Rahul
>
> On Thu, Jan 28, 2016 at 1:39 PM, Jason Altekruse <altekrusejason@gmail.com
> >
> wrote:
>
> > I think it makes sense to wait for Venki's fix of the API, as releasing a
> > version with the change would make reverting/refactoring the API breaking
> > change less useful.
> >
> > Do you have any idea about the scope of fixing the hive native reader, or
> > which change since the 1.4 release broke it? Is someone available to look
> > at it today?
> >
> > On Thu, Jan 28, 2016 at 1:33 PM, rahul challapalli <
> > challapallirahul@gmail.com> wrote:
> >
> > > Just found an issue with hive native parquet reader (DRILL-4323). This
> > is a
> > > regression from 1.4 and in my opinion should be treated as a blocker.
> > >
> > > - Rahul
> > >
> > > On Thu, Jan 28, 2016 at 1:12 PM, Venki Korukanti <
> > > venki.korukanti@gmail.com>
> > > wrote:
> > >
> > > > Sorry I didn't realize I was breaking the public interface API. Let
> me
> > > work
> > > > on a patch to make it backward compatible.
> > > >
> > > > Thanks
> > > > Venki
> > > >
> > > > On Thu, Jan 28, 2016 at 12:54 PM, Steven Phillips <steven@dremio.com
> >
> > > > wrote:
> > > >
> > > > > I just wanted to bring up an issue that I just now discovered, that
> > has
> > > > > caused me a fair amount of grief.
> > > > >
> > > > > https://github.com/apache/drill/pull/300/commits
> > > > >
> > > > > DRILL-4198 changes a user-facing API, and causes StoragePlugins
> that
> > > were
> > > > > compiled against currently released versions of Drill to no longer
> > > > > functional properly. I would prefer that this breaking change be
> > > modified
> > > > > to be backward compatible if possible.
> > > > >
> > > > > On Thu, Jan 28, 2016 at 11:23 AM, Jason Altekruse <
> > > > > altekrusejason@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Aman,
> > > > > >
> > > > > > This is the failure that he was seeing. He figured out that
the
> new
> > > > > > exclusions in jdbc-all were not being respected when the build
> was
> > > run
> > > > > with
> > > > > > an older Maven version, causing the jar size to increase
> > > significantly.
> > > > > He
> > > > > > added an enforcer to make sure the JAR didn't grow unexpectedly.
> > Can
> > > > you
> > > > > > try to update your maven version and re-run the build?
> > > > > >
> > > > > > - Jason
> > > > > >
> > > > > > On Thu, Jan 28, 2016 at 11:18 AM, Aman Sinha <
> asinha@maprtech.com>
> > > > > wrote:
> > > > > >
> > > > > > > Jacques, I am getting the following build failure on the
latest
> > > > master
> > > > > > > branch...is this what you saw for the Apache build ?  My
mvn
> > > version
> > > > > > output
> > > > > > > is shown below.  Should we all be upgrading to a newer
mvn ?
> > > > > > >
> > > > > > >
> > > > > > > [INFO] --- maven-enforcer-plugin:1.3.1:enforce
> > > > > > > (enforce-jdbc-jar-compactness) @ drill-jdbc-all ---
> > > > > > > [WARNING] Rule 0:
> > > org.apache.maven.plugins.enforcer.RequireFilesSize
> > > > > > failed
> > > > > > > with message:
> > > > > > > The file drill-jdbc-all-1.5.0-SNAPSHOT.jar is outside the
> > expected
> > > > size
> > > > > > > range.
> > > > > > >
> > > > > > >                   This is likely due to you adding new
> > dependencies
> > > > to
> > > > > a
> > > > > > > java-exec and not updating the excludes in this module.
This is
> > > > > important
> > > > > > > as it minimizes the size of the dependency of Drill application
> > > > users.
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /Users/asinha/incubator-drill/exec/jdbc-all/target/drill-jdbc-all-1.5.0-SNAPSHOT.jar
> > > > > > > size (44664121) too large. Max. is
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> 20000000/Users/asinha/incubator-drill/exec/jdbc-all/target/drill-jdbc-all-1.5.0-SNAPSHOT.jar
> > > > > > >
> > > > > > >
> > > > > > > Administrators-MacBook-Pro-144:incubator-drill asinha$
mvn
> > > --version
> > > > > > > Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da;
> > > > > 2013-02-19
> > > > > > > 05:51:28-0800)
> > > > > > > Maven home: /opt/local/share/java/maven3
> > > > > > > Java version: 1.7.0_45, vendor: Oracle Corporation
> > > > > > > Java home:
> > > > > > >
> > /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/jre
> > > > > > > Default locale: en_US, platform encoding: UTF-8
> > > > > > > OS name: "mac os x", version: "10.9.5", arch: "x86_64",
family:
> > > "mac"
> > > > > > >
> > > > > > > On Thu, Jan 28, 2016 at 8:20 AM, Jacques Nadeau <
> > > jacques@dremio.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Build back to normal. It looks like the Apache server
was
> using
> > > an
> > > > > old
> > > > > > > > version of Maven. Once I switched to something more
recent,
> the
> > > > build
> > > > > > > > passed.
> > > > > > > >
> > > > > > > > --
> > > > > > > > Jacques Nadeau
> > > > > > > > CTO and Co-Founder, Dremio
> > > > > > > >
> > > > > > > > On Thu, Jan 28, 2016 at 7:02 AM, Jacques Nadeau <
> > > > jacques@dremio.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hmm... this merge caused the Apache build to
fail.
> > > > Investigating...
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Jacques Nadeau
> > > > > > > > > CTO and Co-Founder, Dremio
> > > > > > > > >
> > > > > > > > > On Thu, Jan 28, 2016 at 6:31 AM, Jacques Nadeau
<
> > > > > jacques@dremio.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> I got clean regression runs as well. I've
merged the
> patch.
> > > > > > > > >>
> > > > > > > > >> Jason, you want to start the release process?
> > > > > > > > >>
> > > > > > > > >> --
> > > > > > > > >> Jacques Nadeau
> > > > > > > > >> CTO and Co-Founder, Dremio
> > > > > > > > >>
> > > > > > > > >> On Wed, Jan 27, 2016 at 10:42 PM, Abhishek
Girish <
> > > > > agirish@mapr.com
> > > > > > >
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >>> Had two clean Functional runs. TPC-H
SF100 was also
> > > successful.
> > > > > > > > >>>
> > > > > > > > >>> On Wed, Jan 27, 2016 at 10:07 PM, rahul
challapalli <
> > > > > > > > >>> challapallirahul@gmail.com> wrote:
> > > > > > > > >>>
> > > > > > > > >>> > Kicked off a functional run with
your branch. Will let
> > you
> > > > know
> > > > > > > once
> > > > > > > > it
> > > > > > > > >>> > finishes
> > > > > > > > >>> >
> > > > > > > > >>> > - Rahul
> > > > > > > > >>> >
> > > > > > > > >>> > On Wed, Jan 27, 2016 at 9:56 PM,
Jacques Nadeau <
> > > > > > > jacques@dremio.com>
> > > > > > > > >>> > wrote:
> > > > > > > > >>> >
> > > > > > > > >>> > > 4196 was merged today. I have
an updated patch for
> 4291
> > > > that
> > > > > is
> > > > > > > > >>> ready.
> > > > > > > > >>> > > Unfortunately, it seems that
something isn't working
> > with
> > > > our
> > > > > > > > >>> extended
> > > > > > > > >>> > > tests so I haven't been able
to run an extended
> > > regression.
> > > > > > Unit
> > > > > > > > >>> tests
> > > > > > > > >>> > > pass. Is someone else possibly
able to run a
> regression
> > > > suite
> > > > > > > > against
> > > > > > > > >>> > this
> > > > > > > > >>> > > branch [1] so we can confirm
things look good and
> start
> > > the
> > > > > > > release
> > > > > > > > >>> > > process?
> > > > > > > > >>> > >
> > > > > > > > >>> > > thanks,
> > > > > > > > >>> > > Jacques
> > > > > > > > >>> > >
> > > > > > > > >>> > > [1]
> > https://github.com/jacques-n/drill/tree/DRILL-4291v2
> > > > > > > > >>> > >
> > > > > > > > >>> > > --
> > > > > > > > >>> > > Jacques Nadeau
> > > > > > > > >>> > > CTO and Co-Founder, Dremio
> > > > > > > > >>> > >
> > > > > > > > >>> > > On Mon, Jan 25, 2016 at 11:20
AM, Jacques Nadeau <
> > > > > > > > jacques@dremio.com
> > > > > > > > >>> >
> > > > > > > > >>> > > wrote:
> > > > > > > > >>> > >
> > > > > > > > >>> > > > I think the main things
are 4196 and 4291 should be
> > > > > > completed.
> > > > > > > I
> > > > > > > > >>> know
> > > > > > > > >>> > > Amit
> > > > > > > > >>> > > > was able to reproduce
4196 locally this weekend so
> I
> > > > think
> > > > > > > we're
> > > > > > > > >>> close
> > > > > > > > >>> > on
> > > > > > > > >>> > > > that.
> > > > > > > > >>> > > >
> > > > > > > > >>> > > > --
> > > > > > > > >>> > > > Jacques Nadeau
> > > > > > > > >>> > > > CTO and Co-Founder, Dremio
> > > > > > > > >>> > > >
> > > > > > > > >>> > > > On Mon, Jan 25, 2016 at
10:14 AM, Zelaine Fong <
> > > > > > > > zfong@maprtech.com
> > > > > > > > >>> >
> > > > > > > > >>> > > wrote:
> > > > > > > > >>> > > >
> > > > > > > > >>> > > >> Any updates on this?
 What's blocking us from
> taking
> > > > this
> > > > > > to a
> > > > > > > > >>> vote --
> > > > > > > > >>> > > the
> > > > > > > > >>> > > >> sort merge join issues?
> > > > > > > > >>> > > >>
> > > > > > > > >>> > > >> -- Zelaine
> > > > > > > > >>> > > >>
> > > > > > > > >>> > > >> On Tue, Jan 19, 2016
at 4:18 PM, Jacques Nadeau <
> > > > > > > > >>> jacques@dremio.com>
> > > > > > > > >>> > > >> wrote:
> > > > > > > > >>> > > >>
> > > > > > > > >>> > > >> > Bumping this
thread...
> > > > > > > > >>> > > >> >
> > > > > > > > >>> > > >> > Here are the
issues that were mentioned in this
> > > thread
> > > > > > along
> > > > > > > > >>> with a
> > > > > > > > >>> > > >> > proposed categorization:
> > > > > > > > >>> > > >> >
> > > > > > > > >>> > > >> > Release Blockers
> > > > > > > > >>> > > >> > In-progress Amit
> > > > > > > > >>> https://issues.apache.org/jira/browse/DRILL-4190
> > > > > > > > >>> > > >> > In-progress Amit
> > > > > > > > >>> https://issues.apache.org/jira/browse/DRILL-4196
> > > > > > > > >>> > > >> > Ready to merge
Jacques
> > > > > > > > >>> > > https://issues.apache.org/jira/browse/DRILL-4246
> > > > > > > > >>> > > >> > In-review Jinfeng
> > > > > > > > >>> https://issues.apache.org/jira/browse/DRILL-4256
> > > > > > > > >>> > > >> > In-progress Jacques
> > > > > > > > >>> > https://issues.apache.org/jira/browse/DRILL-4278
> > > > > > > > >>> > > >> > Ready to merge
Laurent
> > > > > > > > >>> > > https://issues.apache.org/jira/browse/DRILL-4285
> > > > > > > > >>> > > >> > Nice to Have
> > > > > > > > >>> > > >> > Open Jason/Hakim
> > > > > > > > >>> https://issues.apache.org/jira/browse/DRILL-4247
> > > > > > > > >>> > > >> > In-progress Jason
> > > > > > > > >>> https://issues.apache.org/jira/browse/DRILL-4203
> > > > > > > > >>> > > >> > Open Jacques
> > > > > > > https://issues.apache.org/jira/browse/DRILL-4266
> > > > > > > > >>> > > >> > Ready to merge
Jacques
> > > > > > > > >>> > > https://issues.apache.org/jira/browse/DRILL-4131
> > > > > > > > >>> > > >> >
> > > > > > > > >>> > > >> > What do others
think? Let's try to get the
> > blockers
> > > > > > wrapped
> > > > > > > up
> > > > > > > > >>> in
> > > > > > > > >>> > the
> > > > > > > > >>> > > >> next
> > > > > > > > >>> > > >> > day or two and
start a release vote...
> > > > > > > > >>> > > >> >
> > > > > > > > >>> > > >> >
> > > > > > > > >>> > > >> >
> > > > > > > > >>> > > >> > --
> > > > > > > > >>> > > >> > Jacques Nadeau
> > > > > > > > >>> > > >> > CTO and Co-Founder,
Dremio
> > > > > > > > >>> > > >> >
> > > > > > > > >>> > > >> > On Mon, Jan 4,
2016 at 1:48 PM, Jason Altekruse
> <
> > > > > > > > >>> > > >> altekrusejason@gmail.com>
> > > > > > > > >>> > > >> > wrote:
> > > > > > > > >>> > > >> >
> > > > > > > > >>> > > >> > > Hello All,
> > > > > > > > >>> > > >> > >
> > > > > > > > >>> > > >> > > With the
allocator changes merged and about a
> > > month
> > > > > > since
> > > > > > > > the
> > > > > > > > >>> last
> > > > > > > > >>> > > >> > release
> > > > > > > > >>> > > >> > > I think
it would be good to start a vote
> soon. I
> > > > would
> > > > > > > like
> > > > > > > > to
> > > > > > > > >>> > > >> volunteer
> > > > > > > > >>> > > >> > to
> > > > > > > > >>> > > >> > > be release
manager.
> > > > > > > > >>> > > >> > >
> > > > > > > > >>> > > >> > > I know that
there were some issues that were
> > > > > identified
> > > > > > > > after
> > > > > > > > >>> the
> > > > > > > > >>> > > >> > transfer
> > > > > > > > >>> > > >> > > patch was
merged. I think that these issues
> > should
> > > > be
> > > > > > > fixed
> > > > > > > > >>> before
> > > > > > > > >>> > > we
> > > > > > > > >>> > > >> > cut a
> > > > > > > > >>> > > >> > > release
candidate.
> > > > > > > > >>> > > >> > >
> > > > > > > > >>> > > >> > > From looking
at the associated JIRAs it looked
> > > like
> > > > > > there
> > > > > > > > was
> > > > > > > > >>> a
> > > > > > > > >>> > > >> possible
> > > > > > > > >>> > > >> > > short term
fix just adjusting the
> > > > > > > max_query_memory_per_node
> > > > > > > > >>> > option,
> > > > > > > > >>> > > >> and
> > > > > > > > >>> > > >> > > some more
involved work to change how we
> > determine
> > > > the
> > > > > > > > correct
> > > > > > > > >>> > time
> > > > > > > > >>> > > to
> > > > > > > > >>> > > >> > > spill during
external sort. I believe it makes
> > > sense
> > > > > to
> > > > > > > make
> > > > > > > > >>> > > external
> > > > > > > > >>> > > >> > sort
> > > > > > > > >>> > > >> > > work well
with the newly improved memory
> > > accounting
> > > > > > before
> > > > > > > > >>> > cutting a
> > > > > > > > >>> > > >> > > release,
but I'm not sure how much work is
> left
> > to
> > > > be
> > > > > > done
> > > > > > > > >>> there.
> > > > > > > > >>> > > [1]
> > > > > > > > >>> > > >> > >
> > > > > > > > >>> > > >> > > Please respond
with your thoughts on a release
> > > soon
> > > > > and
> > > > > > > any
> > > > > > > > >>> JIRAs
> > > > > > > > >>> > > you
> > > > > > > > >>> > > >> > would
> > > > > > > > >>> > > >> > > like to
include in the release.
> > > > > > > > >>> > > >> > >
> > > > > > > > >>> > > >> > > [1] -
> > > > > https://issues.apache.org/jira/browse/DRILL-4243
> > > > > > > > >>> > > >> > >
> > > > > > > > >>> > > >> > > Thanks,
> > > > > > > > >>> > > >> > > Jason
> > > > > > > > >>> > > >> > >
> > > > > > > > >>> > > >> >
> > > > > > > > >>> > > >>
> > > > > > > > >>> > > >
> > > > > > > > >>> > > >
> > > > > > > > >>> > >
> > > > > > > > >>> >
> > > > > > > > >>>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message