drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <ted.dunn...@gmail.com>
Subject Re: First pass at a reference interpreter
Date Wed, 16 Jan 2013 03:22:42 GMT
I would suggest not putting the output in a fixed path.  Most users won't
have write permission by default.

On Tue, Jan 15, 2013 at 9:26 AM, Jacques Nadeau <jacques.drill@gmail.com>wrote:

> Command line running:
>
> cd sandbox/prototype/exec/ref
> mvn exec:java
> -Dexec.mainClass="org.apache.drill.exec.ref.ReferenceInterpreter" && cat
> /opt/data/out.json
> (note, the cat is required because the only current sink is a json writer
> and the plan currently states to write to this location.)
>
> Data input and query files are at:
> sandbox/prototype/exec/ref/src/test/resources/
> >donuts.json is a small sample data file.
> >simple_plan.json is a logical plan.
>
>
>
> On Mon, Jan 14, 2013 at 5:07 PM, Michael Hausenblas <
> michael.hausenblas@gmail.com> wrote:
>
> >
> > > Given that Java6 is public updates EOL next month, I've targeted Java7.
> > > You probably need to update your JDK.
> >
> > Good point. I suppose this should be in big red blinking letters on top
> of
> > the installation instructions.
> >
> > 'mvn verify' is a success now for me ... in case anyone is on MacOS 10,
> > I've used the JDK from [1] and works like a charm.
> >
> > Cheers,
> >                 Michael
> >
> > [1]
> >
> http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html
> >
> > --
> > Michael Hausenblas
> > Ireland, Europe
> > http://mhausenblas.info/
> >
> > On 14 Jan 2013, at 16:37, Jacques Nadeau <jacques.drill@gmail.com>
> wrote:
> >
> > > Given that Java6 is public updates EOL next month, I've targeted Java7.
> > > You probably need to update your JDK.
> > >
> > > Jacques
> > >
> > > On Mon, Jan 14, 2013 at 4:31 PM, Michael Hausenblas <
> > > michael.hausenblas@gmail.com> wrote:
> > >
> > >>
> > >>> Be careful, there are many sharp edges.
> > >>>
> > >>> :)
> > >>
> > >>
> > >> Alright ;)
> > >>
> > >> So here we go: 'mvn validate' works fine for me but then ...
> > >>
> > >> $ mvn verify
> > >>
> > >> [INFO] prototype-parent .................................. SUCCESS
> > [7.166s]
> > >> [INFO] common ............................................ FAILURE
> > >> [17.775s]
> > >> [INFO] contrib-parent .................................... SKIPPED
> > >> [INFO] pom ............................................... SKIPPED
> > >> [INFO] exec .............................................. SKIPPED
> > >> [INFO] java-exec ......................................... SKIPPED
> > >> [INFO] Logical Plan Execution Reference Implementation ... SKIPPED
> > >> [INFO] planner ........................................... SKIPPED
> > >> [INFO] sqlparser ......................................... SKIPPED
> > >>
> > >>
> > >> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to
> > execute
> > >> goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:compile
> > >> (default-compile) on project common: Compilation failure
> > >> Failure executing javac, but could not parse the error:
> > >> javac: invalid target release: 1.7
> > >> Usage: javac <options> <source files>
> > >> use -help for a list of possible options
> > >>
> > >>
> > >>        at
> > >>
> >
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
> > >>        at
> > >>
> >
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> > >>        at
> > >>
> >
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> > >>        at
> > >>
> >
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
> > >>        at
> > >>
> >
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
> > >>        at
> > >>
> >
> org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
> > >>        at
> > >>
> >
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
> > >>        at
> org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:319)
> > >>        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
> > >>        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
> > >>        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
> > >>        at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
> > >>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > >>        at
> > >>
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > >>        at
> > >>
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > >>        at java.lang.reflect.Method.invoke(Method.java:597)
> > >>        at
> > >>
> >
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
> > >>        at
> > >>
> >
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
> > >>        at
> > >>
> >
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
> > >>        at
> > >>
> > org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
> > >> Caused by: org.apache.maven.plugin.CompilationFailureException:
> > >> Compilation failure
> > >> Failure executing javac, but could not parse the error:
> > >> javac: invalid target release: 1.7
> > >> Usage: javac <options> <source files>
> > >> use -help for a list of possible options
> > >>
> > >>
> > >>        at
> > >>
> >
> org.apache.maven.plugin.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:516)
> > >>        at
> > >> org.apache.maven.plugin.CompilerMojo.execute(CompilerMojo.java:114)
> > >>        at
> > >>
> >
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
> > >>        at
> > >>
> >
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
> > >>
> > >>
> > >>
> > >> Cheers,
> > >>                Michael
> > >>
> > >> --
> > >> Michael Hausenblas
> > >> Ireland, Europe
> > >> http://mhausenblas.info/
> > >>
> > >> On 14 Jan 2013, at 16:06, Jacques Nadeau <jacques.drill@gmail.com>
> > wrote:
> > >>
> > >>> Be careful, there are many sharp edges.
> > >>>
> > >>> :)
> > >>>
> > >>> On Mon, Jan 14, 2013 at 4:02 PM, Michael Hausenblas <
> > >>> michael.hausenblas@gmail.com> wrote:
> > >>>
> > >>>>
> > >>>> Cool stuff, Jacques - will give it a shot ASAP!
> > >>>>
> > >>>> Cheers,
> > >>>>               Michael
> > >>>>
> > >>>> --
> > >>>> Michael Hausenblas
> > >>>> Ireland, Europe
> > >>>> http://mhausenblas.info/
> > >>>>
> > >>>> On 14 Jan 2013, at 15:56, Jacques Nadeau <jacques.drill@gmail.com>
> > >> wrote:
> > >>>>
> > >>>>> I've been pulling together a reference logical plan interpreter.
>  I'm
> > >>>>> working with Ted to get it inside the Drill sandbox.  For now,
you
> > can
> > >>>> find
> > >>>>> it on my repo at
> > https://github.com/jacques-n/incubator-drill(prototype
> > >>>>> branch)
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> The goals of the reference interpreter are:
> > >>>>>
> > >>>>>
> > >>>>> - To provide a simple way to run a Logical Plan against some
sample
> > >>>> data
> > >>>>> and get back the expected result
> > >>>>> - Allow work to start on the parsers while we scale up the
> > performance
> > >>>>> and capabilities of the execution engine and optimizer.
> > >>>>> - Allow evaluation work on particular technical approaches
such as
> > >>>>> exploring the impact of hierarchical and schema less data on
query
> > >>>>> evaluation.
> > >>>>>
> > >>>>> These goals do not include performance, memory handling, or
> > >>>>> efficiency.  Currently,
> > >>>>> the interpreter is a single node/thread process.  This will
change
> > >>>> shortly
> > >>>>> so that it also run as a clustered process.
> > >>>>>
> > >>>>> The entry point is inside the /sandbox/prototype/exec/ref module:
> > >>>>> org.apache.drill.exec.ref.ReferenceInterpreter.main();  The
example
> > >>>> program
> > >>>>> utilizes two resources: simple-plan.json and donuts.json and
> outputs
> > >> data
> > >>>>> to /opt/data/out.json.
> > >>>>>
> > >>>>>
> > >>>>> Some of things that 'work'.
> > >>>>>
> > >>>>>
> > >>>>> - Read/write basic json.
> > >>>>> - ROPs (reference operators): Filter, Transform, Group, Aggregate
> > >>>>> (simple), Order, Union.
> > >>>>> - Example aggregate and basic functions including sum, count,
> > >> multiply,
> > >>>>> add, compare, equals.
> > >>>>>
> > >>>>> Basic glossary/concepts (we'll get this on the wiki/javadocs):
> > >>>>>
> > >>>>>
> > >>>>> - LOP: Logical Operator.  An implementation agnostic data flow
> > >> operator
> > >>>>> utilized by the Logical Plan.
> > >>>>> - ROP: Reference Operator: A reference operator implementation
that
> > >>>>> pairs with a LOP.
> > >>>>> - FunctionDefinition: A definition of a particular function.
> > >> Describes
> > >>>>> a set of aliases, an allowable set of input arguments and an
> > interface
> > >>>> that
> > >>>>> will attempt to determine output type.
> > >>>>> - BasicEvaluator: An implementation of a particular non-aggregate
> > >>>>> expression.  Receives a record pointer at creation time. Returns
a
> > >>>>> DataValue.
> > >>>>> - AggregateEvaluator: An implementation of a particular aggregating
> > >>>>> function.  Is provided a record pointer at creation time. 
Expects
> > >>>> regular
> > >>>>> calls to addRecord() followed by a call to eval() which provides
> the
> > >>>>> aggregate value.
> > >>>>> - DataValue: A pointer to a particular data value.  Implementation
> > >>>>> classes includes things like ScalarLong, ScalarBytes,
> SimpleMapValue
> > >>>> and
> > >>>>> SimpleArrayValue.
> > >>>>>
> > >>>>> The standard record iterator utilized between each ROP utilizes
the
> > >>>>> org.apache.drill.exec.ref.RecordIterator interface.  This is
> somewhat
> > >>>>> inspired by the AttributeSource concepts from within the Lucene
> > >> project.
> > >>>>> (I'm planning to extend these concepts all the way to the
> individual
> > >>>>> DataValues.)
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> My next goals are to add tests, finish adding ROPs, add local
and
> > >> remote
> > >>>>> exchange nodes (parallelization), add a bunch of documentation
and
> > >>>> extract
> > >>>>> out the Execution plan as a separate intermediate representation.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> It needs a lot more evaluators to be a true reference interpreter
> (as
> > >>>> well
> > >>>>> as the rest of the ROPs).  The existing ones can be utilized
as
> > >>>> prototypes.
> > >>>>> Anyone interested in ripping through a bunch of additional
> evaluators
> > >> and
> > >>>>> associated FunctionDefinitions?
> > >>>>
> > >>>>
> > >>
> > >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message