drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacques Nadeau <jacq...@dremio.com>
Subject Re: [DISCUSS] Proposal to turn ValueVectors into separate reusable library & project
Date Mon, 09 Nov 2015 01:09:01 GMT
Ok guys,

I took the quiet time directly after the release candidate went out to do
the first phase of componentization. You can see my work at [1].

This set of commits has little functional impact. I've also done my best to
avoid package or file renaming, rather keeping things in their same
packages but in different modules (so that other patches are more easily
applied). There are nine commits in the branch. They break down into three

I've separated the changes out so that it should be reasonably
straightforward to review. The MOVE patches are constrained primarily to
moving files from module to another.

DRILL-3987: (MOVE) Extract key vector, field reader, complex/field wr… …
DRILL-3987: (REFACTOR) Common and Vector modules building. … e390db9
DRILL-3987: (REFACTOR) Working TPCH unit tests … 2cc1d30
DRILL-3987: (MOVE) Extract RPC, memory-base and memory-impl as separa… …
DRILL-3987: (REFACTOR) Extract BoundsChecking check from AssertionUti… …
DRILL-3987: (CLEANUP) Delete unused files 5d596d5
DRILL-3987: (REFACTOR) Remove any parent Drill dependencies for drill… …
DRILL-3987: (MOVE) Move logical expressions and operators out of comm… …
DRILL-3987: (CLEANUP) Final cleanups to get complete working build/di… …

The main goal was to extract a number of separate java-exec submodules.
I've also outlined the modularization in a couple slides at [2]. In those
slides you'll see that there are some orange dependencies that will need to
be removed in a second phase of effort. We also need to decide which
portions of the third slide at [2] would be appropriate as a separate
project versus maintained inside of Drill.

Some of the dependencies will need a finer grained hand to separate. The
biggest remaining is cleaning up VectorDescriptor, MaterializedField,
SerializedField, SchemaPath and FieldReference so that vector can stop
depending on the new drill-logical module.

My preference would be to merge this straight away as the functional impact
is limited and it would be exceedingly difficult to maintain this patch.
This patch set provides a complete set of changes for modularization and
passes all unit tests. I'm running the extended regression suite now to
confirm no impact on those issues. I don't expect any since the only bugs
I've had to track down thus far are drill-module or pom dependency issues.

Let me know your thoughts.

[1] https://github.com/apache/drill/pull/250

Jacques Nadeau
CTO and Co-Founder, Dremio

On Tue, Oct 27, 2015 at 5:59 PM, Jacques Nadeau <jacques@dremio.com> wrote:

> Yes, I've started the umbrella @
> https://issues.apache.org/jira/browse/DRILL-3986
> And the first sub task: extraction poc @
> https://issues.apache.org/jira/browse/DRILL-3987
> I posted some existing materials. I'll start looking at how we can
> extract. Would love others thoughts about how we might slice things. I'll
> post some initial thoughts on the jiras in this regard.
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
> On Tue, Oct 27, 2015 at 5:39 PM, Julian Hyde <jhyde@apache.org> wrote:
>> Jacques, Can you please log the JIRA case you mentioned, and also attach
>> any documentation (e.g. javadoc) you already have.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message