drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aman Sinha <asi...@maprtech.com>
Subject Re: Time for a 1.5 release?
Date Thu, 28 Jan 2016 22:44:08 GMT
I was able to do the build with maven 3.3.9 (after doing 'brew install
maven').  Here are my env variables related to maven:
M2_HOME=/usr/local/Cellar/maven/3.3.9/libexec
M2=/usr/local/Cellar/maven/3.3.9/libexec/bin
MAVEN_OPTS=-Xmx2048m -XX:MaxPermSize=256m


On Thu, Jan 28, 2016 at 2:20 PM, Jacques Nadeau <jacques@dremio.com> wrote:

> Aman, for reference on the Maven version issue. You can see the output of
> the shade plugin between the two Apache Builds. If you search for
> maven-shade-plugin you'll see that build 647 (maven 3.0.5) fails because it
> includes a bunch of hadoop (and other libraries that are listed as being
> excluded). If you look at build 648 (maven 3.3.3), those items are included
> so that the build completes successfully.
>
> https://builds.apache.org/job/drill-scm/647/consoleText
> https://builds.apache.org/job/drill-scm/648/consoleText
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Thu, Jan 28, 2016 at 2:01 PM, rahul challapalli <
> challapallirahul@gmail.com> wrote:
>
> > Jason,
> >
> > As of now I haven't narrowed down the exact commit which caused
> DRILL-4323
> > and I do not know the scope of the fix.
> >
> > - Rahul
> >
> > On Thu, Jan 28, 2016 at 1:39 PM, Jason Altekruse <
> altekrusejason@gmail.com
> > >
> > wrote:
> >
> > > I think it makes sense to wait for Venki's fix of the API, as
> releasing a
> > > version with the change would make reverting/refactoring the API
> breaking
> > > change less useful.
> > >
> > > Do you have any idea about the scope of fixing the hive native reader,
> or
> > > which change since the 1.4 release broke it? Is someone available to
> look
> > > at it today?
> > >
> > > On Thu, Jan 28, 2016 at 1:33 PM, rahul challapalli <
> > > challapallirahul@gmail.com> wrote:
> > >
> > > > Just found an issue with hive native parquet reader (DRILL-4323).
> This
> > > is a
> > > > regression from 1.4 and in my opinion should be treated as a blocker.
> > > >
> > > > - Rahul
> > > >
> > > > On Thu, Jan 28, 2016 at 1:12 PM, Venki Korukanti <
> > > > venki.korukanti@gmail.com>
> > > > wrote:
> > > >
> > > > > Sorry I didn't realize I was breaking the public interface API. Let
> > me
> > > > work
> > > > > on a patch to make it backward compatible.
> > > > >
> > > > > Thanks
> > > > > Venki
> > > > >
> > > > > On Thu, Jan 28, 2016 at 12:54 PM, Steven Phillips <
> steven@dremio.com
> > >
> > > > > wrote:
> > > > >
> > > > > > I just wanted to bring up an issue that I just now discovered,
> that
> > > has
> > > > > > caused me a fair amount of grief.
> > > > > >
> > > > > > https://github.com/apache/drill/pull/300/commits
> > > > > >
> > > > > > DRILL-4198 changes a user-facing API, and causes StoragePlugins
> > that
> > > > were
> > > > > > compiled against currently released versions of Drill to no
> longer
> > > > > > functional properly. I would prefer that this breaking change
be
> > > > modified
> > > > > > to be backward compatible if possible.
> > > > > >
> > > > > > On Thu, Jan 28, 2016 at 11:23 AM, Jason Altekruse <
> > > > > > altekrusejason@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Aman,
> > > > > > >
> > > > > > > This is the failure that he was seeing. He figured out
that the
> > new
> > > > > > > exclusions in jdbc-all were not being respected when the
build
> > was
> > > > run
> > > > > > with
> > > > > > > an older Maven version, causing the jar size to increase
> > > > significantly.
> > > > > > He
> > > > > > > added an enforcer to make sure the JAR didn't grow
> unexpectedly.
> > > Can
> > > > > you
> > > > > > > try to update your maven version and re-run the build?
> > > > > > >
> > > > > > > - Jason
> > > > > > >
> > > > > > > On Thu, Jan 28, 2016 at 11:18 AM, Aman Sinha <
> > asinha@maprtech.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Jacques, I am getting the following build failure
on the
> latest
> > > > > master
> > > > > > > > branch...is this what you saw for the Apache build
?  My mvn
> > > > version
> > > > > > > output
> > > > > > > > is shown below.  Should we all be upgrading to a newer
mvn ?
> > > > > > > >
> > > > > > > >
> > > > > > > > [INFO] --- maven-enforcer-plugin:1.3.1:enforce
> > > > > > > > (enforce-jdbc-jar-compactness) @ drill-jdbc-all ---
> > > > > > > > [WARNING] Rule 0:
> > > > org.apache.maven.plugins.enforcer.RequireFilesSize
> > > > > > > failed
> > > > > > > > with message:
> > > > > > > > The file drill-jdbc-all-1.5.0-SNAPSHOT.jar is outside
the
> > > expected
> > > > > size
> > > > > > > > range.
> > > > > > > >
> > > > > > > >                   This is likely due to you adding
new
> > > dependencies
> > > > > to
> > > > > > a
> > > > > > > > java-exec and not updating the excludes in this module.
This
> is
> > > > > > important
> > > > > > > > as it minimizes the size of the dependency of Drill
> application
> > > > > users.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /Users/asinha/incubator-drill/exec/jdbc-all/target/drill-jdbc-all-1.5.0-SNAPSHOT.jar
> > > > > > > > size (44664121) too large. Max. is
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> 20000000/Users/asinha/incubator-drill/exec/jdbc-all/target/drill-jdbc-all-1.5.0-SNAPSHOT.jar
> > > > > > > >
> > > > > > > >
> > > > > > > > Administrators-MacBook-Pro-144:incubator-drill asinha$
mvn
> > > > --version
> > > > > > > > Apache Maven 3.0.5
> (r01de14724cdef164cd33c7c8c2fe155faf9602da;
> > > > > > 2013-02-19
> > > > > > > > 05:51:28-0800)
> > > > > > > > Maven home: /opt/local/share/java/maven3
> > > > > > > > Java version: 1.7.0_45, vendor: Oracle Corporation
> > > > > > > > Java home:
> > > > > > > >
> > > /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/jre
> > > > > > > > Default locale: en_US, platform encoding: UTF-8
> > > > > > > > OS name: "mac os x", version: "10.9.5", arch: "x86_64",
> family:
> > > > "mac"
> > > > > > > >
> > > > > > > > On Thu, Jan 28, 2016 at 8:20 AM, Jacques Nadeau <
> > > > jacques@dremio.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Build back to normal. It looks like the Apache
server was
> > using
> > > > an
> > > > > > old
> > > > > > > > > version of Maven. Once I switched to something
more recent,
> > the
> > > > > build
> > > > > > > > > passed.
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Jacques Nadeau
> > > > > > > > > CTO and Co-Founder, Dremio
> > > > > > > > >
> > > > > > > > > On Thu, Jan 28, 2016 at 7:02 AM, Jacques Nadeau
<
> > > > > jacques@dremio.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hmm... this merge caused the Apache build
to fail.
> > > > > Investigating...
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Jacques Nadeau
> > > > > > > > > > CTO and Co-Founder, Dremio
> > > > > > > > > >
> > > > > > > > > > On Thu, Jan 28, 2016 at 6:31 AM, Jacques
Nadeau <
> > > > > > jacques@dremio.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> I got clean regression runs as well.
I've merged the
> > patch.
> > > > > > > > > >>
> > > > > > > > > >> Jason, you want to start the release
process?
> > > > > > > > > >>
> > > > > > > > > >> --
> > > > > > > > > >> Jacques Nadeau
> > > > > > > > > >> CTO and Co-Founder, Dremio
> > > > > > > > > >>
> > > > > > > > > >> On Wed, Jan 27, 2016 at 10:42 PM, Abhishek
Girish <
> > > > > > agirish@mapr.com
> > > > > > > >
> > > > > > > > > >> wrote:
> > > > > > > > > >>
> > > > > > > > > >>> Had two clean Functional runs. TPC-H
SF100 was also
> > > > successful.
> > > > > > > > > >>>
> > > > > > > > > >>> On Wed, Jan 27, 2016 at 10:07 PM,
rahul challapalli <
> > > > > > > > > >>> challapallirahul@gmail.com> wrote:
> > > > > > > > > >>>
> > > > > > > > > >>> > Kicked off a functional run
with your branch. Will
> let
> > > you
> > > > > know
> > > > > > > > once
> > > > > > > > > it
> > > > > > > > > >>> > finishes
> > > > > > > > > >>> >
> > > > > > > > > >>> > - Rahul
> > > > > > > > > >>> >
> > > > > > > > > >>> > On Wed, Jan 27, 2016 at 9:56
PM, Jacques Nadeau <
> > > > > > > > jacques@dremio.com>
> > > > > > > > > >>> > wrote:
> > > > > > > > > >>> >
> > > > > > > > > >>> > > 4196 was merged today.
I have an updated patch for
> > 4291
> > > > > that
> > > > > > is
> > > > > > > > > >>> ready.
> > > > > > > > > >>> > > Unfortunately, it seems
that something isn't
> working
> > > with
> > > > > our
> > > > > > > > > >>> extended
> > > > > > > > > >>> > > tests so I haven't been
able to run an extended
> > > > regression.
> > > > > > > Unit
> > > > > > > > > >>> tests
> > > > > > > > > >>> > > pass. Is someone else
possibly able to run a
> > regression
> > > > > suite
> > > > > > > > > against
> > > > > > > > > >>> > this
> > > > > > > > > >>> > > branch [1] so we can confirm
things look good and
> > start
> > > > the
> > > > > > > > release
> > > > > > > > > >>> > > process?
> > > > > > > > > >>> > >
> > > > > > > > > >>> > > thanks,
> > > > > > > > > >>> > > Jacques
> > > > > > > > > >>> > >
> > > > > > > > > >>> > > [1]
> > > https://github.com/jacques-n/drill/tree/DRILL-4291v2
> > > > > > > > > >>> > >
> > > > > > > > > >>> > > --
> > > > > > > > > >>> > > Jacques Nadeau
> > > > > > > > > >>> > > CTO and Co-Founder, Dremio
> > > > > > > > > >>> > >
> > > > > > > > > >>> > > On Mon, Jan 25, 2016 at
11:20 AM, Jacques Nadeau <
> > > > > > > > > jacques@dremio.com
> > > > > > > > > >>> >
> > > > > > > > > >>> > > wrote:
> > > > > > > > > >>> > >
> > > > > > > > > >>> > > > I think the main
things are 4196 and 4291 should
> be
> > > > > > > completed.
> > > > > > > > I
> > > > > > > > > >>> know
> > > > > > > > > >>> > > Amit
> > > > > > > > > >>> > > > was able to reproduce
4196 locally this weekend
> so
> > I
> > > > > think
> > > > > > > > we're
> > > > > > > > > >>> close
> > > > > > > > > >>> > on
> > > > > > > > > >>> > > > that.
> > > > > > > > > >>> > > >
> > > > > > > > > >>> > > > --
> > > > > > > > > >>> > > > Jacques Nadeau
> > > > > > > > > >>> > > > CTO and Co-Founder,
Dremio
> > > > > > > > > >>> > > >
> > > > > > > > > >>> > > > On Mon, Jan 25, 2016
at 10:14 AM, Zelaine Fong <
> > > > > > > > > zfong@maprtech.com
> > > > > > > > > >>> >
> > > > > > > > > >>> > > wrote:
> > > > > > > > > >>> > > >
> > > > > > > > > >>> > > >> Any updates on
this?  What's blocking us from
> > taking
> > > > > this
> > > > > > > to a
> > > > > > > > > >>> vote --
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > >> sort merge join
issues?
> > > > > > > > > >>> > > >>
> > > > > > > > > >>> > > >> -- Zelaine
> > > > > > > > > >>> > > >>
> > > > > > > > > >>> > > >> On Tue, Jan 19,
2016 at 4:18 PM, Jacques Nadeau
> <
> > > > > > > > > >>> jacques@dremio.com>
> > > > > > > > > >>> > > >> wrote:
> > > > > > > > > >>> > > >>
> > > > > > > > > >>> > > >> > Bumping
this thread...
> > > > > > > > > >>> > > >> >
> > > > > > > > > >>> > > >> > Here are
the issues that were mentioned in
> this
> > > > thread
> > > > > > > along
> > > > > > > > > >>> with a
> > > > > > > > > >>> > > >> > proposed
categorization:
> > > > > > > > > >>> > > >> >
> > > > > > > > > >>> > > >> > Release
Blockers
> > > > > > > > > >>> > > >> > In-progress
Amit
> > > > > > > > > >>> https://issues.apache.org/jira/browse/DRILL-4190
> > > > > > > > > >>> > > >> > In-progress
Amit
> > > > > > > > > >>> https://issues.apache.org/jira/browse/DRILL-4196
> > > > > > > > > >>> > > >> > Ready to
merge Jacques
> > > > > > > > > >>> > > https://issues.apache.org/jira/browse/DRILL-4246
> > > > > > > > > >>> > > >> > In-review
Jinfeng
> > > > > > > > > >>> https://issues.apache.org/jira/browse/DRILL-4256
> > > > > > > > > >>> > > >> > In-progress
Jacques
> > > > > > > > > >>> > https://issues.apache.org/jira/browse/DRILL-4278
> > > > > > > > > >>> > > >> > Ready to
merge Laurent
> > > > > > > > > >>> > > https://issues.apache.org/jira/browse/DRILL-4285
> > > > > > > > > >>> > > >> > Nice to
Have
> > > > > > > > > >>> > > >> > Open Jason/Hakim
> > > > > > > > > >>> https://issues.apache.org/jira/browse/DRILL-4247
> > > > > > > > > >>> > > >> > In-progress
Jason
> > > > > > > > > >>> https://issues.apache.org/jira/browse/DRILL-4203
> > > > > > > > > >>> > > >> > Open Jacques
> > > > > > > > https://issues.apache.org/jira/browse/DRILL-4266
> > > > > > > > > >>> > > >> > Ready to
merge Jacques
> > > > > > > > > >>> > > https://issues.apache.org/jira/browse/DRILL-4131
> > > > > > > > > >>> > > >> >
> > > > > > > > > >>> > > >> > What do
others think? Let's try to get the
> > > blockers
> > > > > > > wrapped
> > > > > > > > up
> > > > > > > > > >>> in
> > > > > > > > > >>> > the
> > > > > > > > > >>> > > >> next
> > > > > > > > > >>> > > >> > day or two
and start a release vote...
> > > > > > > > > >>> > > >> >
> > > > > > > > > >>> > > >> >
> > > > > > > > > >>> > > >> >
> > > > > > > > > >>> > > >> > --
> > > > > > > > > >>> > > >> > Jacques
Nadeau
> > > > > > > > > >>> > > >> > CTO and
Co-Founder, Dremio
> > > > > > > > > >>> > > >> >
> > > > > > > > > >>> > > >> > On Mon,
Jan 4, 2016 at 1:48 PM, Jason
> Altekruse
> > <
> > > > > > > > > >>> > > >> altekrusejason@gmail.com>
> > > > > > > > > >>> > > >> > wrote:
> > > > > > > > > >>> > > >> >
> > > > > > > > > >>> > > >> > > Hello
All,
> > > > > > > > > >>> > > >> > >
> > > > > > > > > >>> > > >> > > With
the allocator changes merged and about
> a
> > > > month
> > > > > > > since
> > > > > > > > > the
> > > > > > > > > >>> last
> > > > > > > > > >>> > > >> > release
> > > > > > > > > >>> > > >> > > I think
it would be good to start a vote
> > soon. I
> > > > > would
> > > > > > > > like
> > > > > > > > > to
> > > > > > > > > >>> > > >> volunteer
> > > > > > > > > >>> > > >> > to
> > > > > > > > > >>> > > >> > > be
release manager.
> > > > > > > > > >>> > > >> > >
> > > > > > > > > >>> > > >> > > I know
that there were some issues that were
> > > > > > identified
> > > > > > > > > after
> > > > > > > > > >>> the
> > > > > > > > > >>> > > >> > transfer
> > > > > > > > > >>> > > >> > > patch
was merged. I think that these issues
> > > should
> > > > > be
> > > > > > > > fixed
> > > > > > > > > >>> before
> > > > > > > > > >>> > > we
> > > > > > > > > >>> > > >> > cut a
> > > > > > > > > >>> > > >> > > release
candidate.
> > > > > > > > > >>> > > >> > >
> > > > > > > > > >>> > > >> > > From
looking at the associated JIRAs it
> looked
> > > > like
> > > > > > > there
> > > > > > > > > was
> > > > > > > > > >>> a
> > > > > > > > > >>> > > >> possible
> > > > > > > > > >>> > > >> > > short
term fix just adjusting the
> > > > > > > > max_query_memory_per_node
> > > > > > > > > >>> > option,
> > > > > > > > > >>> > > >> and
> > > > > > > > > >>> > > >> > > some
more involved work to change how we
> > > determine
> > > > > the
> > > > > > > > > correct
> > > > > > > > > >>> > time
> > > > > > > > > >>> > > to
> > > > > > > > > >>> > > >> > > spill
during external sort. I believe it
> makes
> > > > sense
> > > > > > to
> > > > > > > > make
> > > > > > > > > >>> > > external
> > > > > > > > > >>> > > >> > sort
> > > > > > > > > >>> > > >> > > work
well with the newly improved memory
> > > > accounting
> > > > > > > before
> > > > > > > > > >>> > cutting a
> > > > > > > > > >>> > > >> > > release,
but I'm not sure how much work is
> > left
> > > to
> > > > > be
> > > > > > > done
> > > > > > > > > >>> there.
> > > > > > > > > >>> > > [1]
> > > > > > > > > >>> > > >> > >
> > > > > > > > > >>> > > >> > > Please
respond with your thoughts on a
> release
> > > > soon
> > > > > > and
> > > > > > > > any
> > > > > > > > > >>> JIRAs
> > > > > > > > > >>> > > you
> > > > > > > > > >>> > > >> > would
> > > > > > > > > >>> > > >> > > like
to include in the release.
> > > > > > > > > >>> > > >> > >
> > > > > > > > > >>> > > >> > > [1]
-
> > > > > > https://issues.apache.org/jira/browse/DRILL-4243
> > > > > > > > > >>> > > >> > >
> > > > > > > > > >>> > > >> > > Thanks,
> > > > > > > > > >>> > > >> > > Jason
> > > > > > > > > >>> > > >> > >
> > > > > > > > > >>> > > >> >
> > > > > > > > > >>> > > >>
> > > > > > > > > >>> > > >
> > > > > > > > > >>> > > >
> > > > > > > > > >>> > >
> > > > > > > > > >>> >
> > > > > > > > > >>>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message