drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Altekruse <altekruseja...@gmail.com>
Subject Re: Time for a 1.5 release?
Date Thu, 28 Jan 2016 21:39:57 GMT
I think it makes sense to wait for Venki's fix of the API, as releasing a
version with the change would make reverting/refactoring the API breaking
change less useful.

Do you have any idea about the scope of fixing the hive native reader, or
which change since the 1.4 release broke it? Is someone available to look
at it today?

On Thu, Jan 28, 2016 at 1:33 PM, rahul challapalli <
challapallirahul@gmail.com> wrote:

> Just found an issue with hive native parquet reader (DRILL-4323). This is a
> regression from 1.4 and in my opinion should be treated as a blocker.
>
> - Rahul
>
> On Thu, Jan 28, 2016 at 1:12 PM, Venki Korukanti <
> venki.korukanti@gmail.com>
> wrote:
>
> > Sorry I didn't realize I was breaking the public interface API. Let me
> work
> > on a patch to make it backward compatible.
> >
> > Thanks
> > Venki
> >
> > On Thu, Jan 28, 2016 at 12:54 PM, Steven Phillips <steven@dremio.com>
> > wrote:
> >
> > > I just wanted to bring up an issue that I just now discovered, that has
> > > caused me a fair amount of grief.
> > >
> > > https://github.com/apache/drill/pull/300/commits
> > >
> > > DRILL-4198 changes a user-facing API, and causes StoragePlugins that
> were
> > > compiled against currently released versions of Drill to no longer
> > > functional properly. I would prefer that this breaking change be
> modified
> > > to be backward compatible if possible.
> > >
> > > On Thu, Jan 28, 2016 at 11:23 AM, Jason Altekruse <
> > > altekrusejason@gmail.com>
> > > wrote:
> > >
> > > > Hi Aman,
> > > >
> > > > This is the failure that he was seeing. He figured out that the new
> > > > exclusions in jdbc-all were not being respected when the build was
> run
> > > with
> > > > an older Maven version, causing the jar size to increase
> significantly.
> > > He
> > > > added an enforcer to make sure the JAR didn't grow unexpectedly. Can
> > you
> > > > try to update your maven version and re-run the build?
> > > >
> > > > - Jason
> > > >
> > > > On Thu, Jan 28, 2016 at 11:18 AM, Aman Sinha <asinha@maprtech.com>
> > > wrote:
> > > >
> > > > > Jacques, I am getting the following build failure on the latest
> > master
> > > > > branch...is this what you saw for the Apache build ?  My mvn
> version
> > > > output
> > > > > is shown below.  Should we all be upgrading to a newer mvn ?
> > > > >
> > > > >
> > > > > [INFO] --- maven-enforcer-plugin:1.3.1:enforce
> > > > > (enforce-jdbc-jar-compactness) @ drill-jdbc-all ---
> > > > > [WARNING] Rule 0:
> org.apache.maven.plugins.enforcer.RequireFilesSize
> > > > failed
> > > > > with message:
> > > > > The file drill-jdbc-all-1.5.0-SNAPSHOT.jar is outside the expected
> > size
> > > > > range.
> > > > >
> > > > >                   This is likely due to you adding new dependencies
> > to
> > > a
> > > > > java-exec and not updating the excludes in this module. This is
> > > important
> > > > > as it minimizes the size of the dependency of Drill application
> > users.
> > > > >
> > > > >
> > > >
> > >
> >
> /Users/asinha/incubator-drill/exec/jdbc-all/target/drill-jdbc-all-1.5.0-SNAPSHOT.jar
> > > > > size (44664121) too large. Max. is
> > > > >
> > > > >
> > > >
> > >
> >
> 20000000/Users/asinha/incubator-drill/exec/jdbc-all/target/drill-jdbc-all-1.5.0-SNAPSHOT.jar
> > > > >
> > > > >
> > > > > Administrators-MacBook-Pro-144:incubator-drill asinha$ mvn
> --version
> > > > > Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da;
> > > 2013-02-19
> > > > > 05:51:28-0800)
> > > > > Maven home: /opt/local/share/java/maven3
> > > > > Java version: 1.7.0_45, vendor: Oracle Corporation
> > > > > Java home:
> > > > > /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/jre
> > > > > Default locale: en_US, platform encoding: UTF-8
> > > > > OS name: "mac os x", version: "10.9.5", arch: "x86_64", family:
> "mac"
> > > > >
> > > > > On Thu, Jan 28, 2016 at 8:20 AM, Jacques Nadeau <
> jacques@dremio.com>
> > > > > wrote:
> > > > >
> > > > > > Build back to normal. It looks like the Apache server was using
> an
> > > old
> > > > > > version of Maven. Once I switched to something more recent,
the
> > build
> > > > > > passed.
> > > > > >
> > > > > > --
> > > > > > Jacques Nadeau
> > > > > > CTO and Co-Founder, Dremio
> > > > > >
> > > > > > On Thu, Jan 28, 2016 at 7:02 AM, Jacques Nadeau <
> > jacques@dremio.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hmm... this merge caused the Apache build to fail.
> > Investigating...
> > > > > > >
> > > > > > > --
> > > > > > > Jacques Nadeau
> > > > > > > CTO and Co-Founder, Dremio
> > > > > > >
> > > > > > > On Thu, Jan 28, 2016 at 6:31 AM, Jacques Nadeau <
> > > jacques@dremio.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> I got clean regression runs as well. I've merged the
patch.
> > > > > > >>
> > > > > > >> Jason, you want to start the release process?
> > > > > > >>
> > > > > > >> --
> > > > > > >> Jacques Nadeau
> > > > > > >> CTO and Co-Founder, Dremio
> > > > > > >>
> > > > > > >> On Wed, Jan 27, 2016 at 10:42 PM, Abhishek Girish <
> > > agirish@mapr.com
> > > > >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> Had two clean Functional runs. TPC-H SF100 was
also
> successful.
> > > > > > >>>
> > > > > > >>> On Wed, Jan 27, 2016 at 10:07 PM, rahul challapalli
<
> > > > > > >>> challapallirahul@gmail.com> wrote:
> > > > > > >>>
> > > > > > >>> > Kicked off a functional run with your branch.
Will let you
> > know
> > > > > once
> > > > > > it
> > > > > > >>> > finishes
> > > > > > >>> >
> > > > > > >>> > - Rahul
> > > > > > >>> >
> > > > > > >>> > On Wed, Jan 27, 2016 at 9:56 PM, Jacques Nadeau
<
> > > > > jacques@dremio.com>
> > > > > > >>> > wrote:
> > > > > > >>> >
> > > > > > >>> > > 4196 was merged today. I have an updated
patch for 4291
> > that
> > > is
> > > > > > >>> ready.
> > > > > > >>> > > Unfortunately, it seems that something
isn't working with
> > our
> > > > > > >>> extended
> > > > > > >>> > > tests so I haven't been able to run an
extended
> regression.
> > > > Unit
> > > > > > >>> tests
> > > > > > >>> > > pass. Is someone else possibly able to
run a regression
> > suite
> > > > > > against
> > > > > > >>> > this
> > > > > > >>> > > branch [1] so we can confirm things look
good and start
> the
> > > > > release
> > > > > > >>> > > process?
> > > > > > >>> > >
> > > > > > >>> > > thanks,
> > > > > > >>> > > Jacques
> > > > > > >>> > >
> > > > > > >>> > > [1] https://github.com/jacques-n/drill/tree/DRILL-4291v2
> > > > > > >>> > >
> > > > > > >>> > > --
> > > > > > >>> > > Jacques Nadeau
> > > > > > >>> > > CTO and Co-Founder, Dremio
> > > > > > >>> > >
> > > > > > >>> > > On Mon, Jan 25, 2016 at 11:20 AM, Jacques
Nadeau <
> > > > > > jacques@dremio.com
> > > > > > >>> >
> > > > > > >>> > > wrote:
> > > > > > >>> > >
> > > > > > >>> > > > I think the main things are 4196
and 4291 should be
> > > > completed.
> > > > > I
> > > > > > >>> know
> > > > > > >>> > > Amit
> > > > > > >>> > > > was able to reproduce 4196 locally
this weekend so I
> > think
> > > > > we're
> > > > > > >>> close
> > > > > > >>> > on
> > > > > > >>> > > > that.
> > > > > > >>> > > >
> > > > > > >>> > > > --
> > > > > > >>> > > > Jacques Nadeau
> > > > > > >>> > > > CTO and Co-Founder, Dremio
> > > > > > >>> > > >
> > > > > > >>> > > > On Mon, Jan 25, 2016 at 10:14 AM,
Zelaine Fong <
> > > > > > zfong@maprtech.com
> > > > > > >>> >
> > > > > > >>> > > wrote:
> > > > > > >>> > > >
> > > > > > >>> > > >> Any updates on this?  What's
blocking us from taking
> > this
> > > > to a
> > > > > > >>> vote --
> > > > > > >>> > > the
> > > > > > >>> > > >> sort merge join issues?
> > > > > > >>> > > >>
> > > > > > >>> > > >> -- Zelaine
> > > > > > >>> > > >>
> > > > > > >>> > > >> On Tue, Jan 19, 2016 at 4:18
PM, Jacques Nadeau <
> > > > > > >>> jacques@dremio.com>
> > > > > > >>> > > >> wrote:
> > > > > > >>> > > >>
> > > > > > >>> > > >> > Bumping this thread...
> > > > > > >>> > > >> >
> > > > > > >>> > > >> > Here are the issues that
were mentioned in this
> thread
> > > > along
> > > > > > >>> with a
> > > > > > >>> > > >> > proposed categorization:
> > > > > > >>> > > >> >
> > > > > > >>> > > >> > Release Blockers
> > > > > > >>> > > >> > In-progress Amit
> > > > > > >>> https://issues.apache.org/jira/browse/DRILL-4190
> > > > > > >>> > > >> > In-progress Amit
> > > > > > >>> https://issues.apache.org/jira/browse/DRILL-4196
> > > > > > >>> > > >> > Ready to merge Jacques
> > > > > > >>> > > https://issues.apache.org/jira/browse/DRILL-4246
> > > > > > >>> > > >> > In-review Jinfeng
> > > > > > >>> https://issues.apache.org/jira/browse/DRILL-4256
> > > > > > >>> > > >> > In-progress Jacques
> > > > > > >>> > https://issues.apache.org/jira/browse/DRILL-4278
> > > > > > >>> > > >> > Ready to merge Laurent
> > > > > > >>> > > https://issues.apache.org/jira/browse/DRILL-4285
> > > > > > >>> > > >> > Nice to Have
> > > > > > >>> > > >> > Open Jason/Hakim
> > > > > > >>> https://issues.apache.org/jira/browse/DRILL-4247
> > > > > > >>> > > >> > In-progress Jason
> > > > > > >>> https://issues.apache.org/jira/browse/DRILL-4203
> > > > > > >>> > > >> > Open Jacques
> > > > > https://issues.apache.org/jira/browse/DRILL-4266
> > > > > > >>> > > >> > Ready to merge Jacques
> > > > > > >>> > > https://issues.apache.org/jira/browse/DRILL-4131
> > > > > > >>> > > >> >
> > > > > > >>> > > >> > What do others think? Let's
try to get the blockers
> > > > wrapped
> > > > > up
> > > > > > >>> in
> > > > > > >>> > the
> > > > > > >>> > > >> next
> > > > > > >>> > > >> > day or two and start a
release vote...
> > > > > > >>> > > >> >
> > > > > > >>> > > >> >
> > > > > > >>> > > >> >
> > > > > > >>> > > >> > --
> > > > > > >>> > > >> > Jacques Nadeau
> > > > > > >>> > > >> > CTO and Co-Founder, Dremio
> > > > > > >>> > > >> >
> > > > > > >>> > > >> > On Mon, Jan 4, 2016 at
1:48 PM, Jason Altekruse <
> > > > > > >>> > > >> altekrusejason@gmail.com>
> > > > > > >>> > > >> > wrote:
> > > > > > >>> > > >> >
> > > > > > >>> > > >> > > Hello All,
> > > > > > >>> > > >> > >
> > > > > > >>> > > >> > > With the allocator
changes merged and about a
> month
> > > > since
> > > > > > the
> > > > > > >>> last
> > > > > > >>> > > >> > release
> > > > > > >>> > > >> > > I think it would be
good to start a vote soon. I
> > would
> > > > > like
> > > > > > to
> > > > > > >>> > > >> volunteer
> > > > > > >>> > > >> > to
> > > > > > >>> > > >> > > be release manager.
> > > > > > >>> > > >> > >
> > > > > > >>> > > >> > > I know that there
were some issues that were
> > > identified
> > > > > > after
> > > > > > >>> the
> > > > > > >>> > > >> > transfer
> > > > > > >>> > > >> > > patch was merged.
I think that these issues should
> > be
> > > > > fixed
> > > > > > >>> before
> > > > > > >>> > > we
> > > > > > >>> > > >> > cut a
> > > > > > >>> > > >> > > release candidate.
> > > > > > >>> > > >> > >
> > > > > > >>> > > >> > > From looking at the
associated JIRAs it looked
> like
> > > > there
> > > > > > was
> > > > > > >>> a
> > > > > > >>> > > >> possible
> > > > > > >>> > > >> > > short term fix just
adjusting the
> > > > > max_query_memory_per_node
> > > > > > >>> > option,
> > > > > > >>> > > >> and
> > > > > > >>> > > >> > > some more involved
work to change how we determine
> > the
> > > > > > correct
> > > > > > >>> > time
> > > > > > >>> > > to
> > > > > > >>> > > >> > > spill during external
sort. I believe it makes
> sense
> > > to
> > > > > make
> > > > > > >>> > > external
> > > > > > >>> > > >> > sort
> > > > > > >>> > > >> > > work well with the
newly improved memory
> accounting
> > > > before
> > > > > > >>> > cutting a
> > > > > > >>> > > >> > > release, but I'm not
sure how much work is left to
> > be
> > > > done
> > > > > > >>> there.
> > > > > > >>> > > [1]
> > > > > > >>> > > >> > >
> > > > > > >>> > > >> > > Please respond with
your thoughts on a release
> soon
> > > and
> > > > > any
> > > > > > >>> JIRAs
> > > > > > >>> > > you
> > > > > > >>> > > >> > would
> > > > > > >>> > > >> > > like to include in
the release.
> > > > > > >>> > > >> > >
> > > > > > >>> > > >> > > [1] -
> > > https://issues.apache.org/jira/browse/DRILL-4243
> > > > > > >>> > > >> > >
> > > > > > >>> > > >> > > Thanks,
> > > > > > >>> > > >> > > Jason
> > > > > > >>> > > >> > >
> > > > > > >>> > > >> >
> > > > > > >>> > > >>
> > > > > > >>> > > >
> > > > > > >>> > > >
> > > > > > >>> > >
> > > > > > >>> >
> > > > > > >>>
> > > > > > >>
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message