From drill-dev-return-902-apmail-incubator-drill-dev-archive=incubator.apache.org@incubator.apache.org Tue Jan 15 17:26:31 2013 Return-Path: X-Original-To: apmail-incubator-drill-dev-archive@minotaur.apache.org Delivered-To: apmail-incubator-drill-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A71DFEF0E for ; Tue, 15 Jan 2013 17:26:31 +0000 (UTC) Received: (qmail 69938 invoked by uid 500); 15 Jan 2013 17:26:30 -0000 Delivered-To: apmail-incubator-drill-dev-archive@incubator.apache.org Received: (qmail 69734 invoked by uid 500); 15 Jan 2013 17:26:29 -0000 Mailing-List: contact drill-dev-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: drill-dev@incubator.apache.org Delivered-To: mailing list drill-dev@incubator.apache.org Received: (qmail 69716 invoked by uid 99); 15 Jan 2013 17:26:29 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jan 2013 17:26:29 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of jacques.drill@gmail.com designates 209.85.220.169 as permitted sender) Received: from [209.85.220.169] (HELO mail-vc0-f169.google.com) (209.85.220.169) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Jan 2013 17:26:24 +0000 Received: by mail-vc0-f169.google.com with SMTP id gb23so381660vcb.28 for ; Tue, 15 Jan 2013 09:26:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=bYWnI0HTLk/uIhiNQb37NWm2uJ0qTnqzZpYTjyXFC2s=; b=vaabgAhmRkodEryD4+sanWzJVWqE9CXRshA4QIOQMNI4Zb2OlTRCX+U+2hj3I/texN //5t4ul8S2833RiTVScqhRIFqvH0ajivejyVhUinlIIz3nqPcTUUwrpFPxsXEVA3aEjU pBU4g9VnOKnLuKjInBptljVViTrX8A7M0aAo4OSBsr/JsJecwOCY1JRYdlRZmYrGTsor gw9CvhwO+8gjxcIRukQncDMXZ4q3PYowvcqstNC6FDQSIJ/m3O6KFm0V5j2JKsLjOEZe nDU8a0vr6V5uFqmBfktSNqXb2PspJgUFQ3P0R6ASpZ3JnZjj1BP3fEwsuJLlPGDBohIy 1/VA== MIME-Version: 1.0 Received: by 10.52.26.229 with SMTP id o5mr92503681vdg.66.1358270763963; Tue, 15 Jan 2013 09:26:03 -0800 (PST) Received: by 10.58.218.169 with HTTP; Tue, 15 Jan 2013 09:26:03 -0800 (PST) In-Reply-To: <840039C8-98C9-4910-A2C2-C9B022889504@gmail.com> References: <4A5CC19B-C9DD-4E99-BBEA-8DC9DF8BBEEF@gmail.com> <0E34A6E6-F070-4E5A-924A-53AEF80F9608@gmail.com> <840039C8-98C9-4910-A2C2-C9B022889504@gmail.com> Date: Tue, 15 Jan 2013 09:26:03 -0800 Message-ID: Subject: Re: First pass at a reference interpreter From: Jacques Nadeau To: drill-dev@incubator.apache.org Content-Type: multipart/alternative; boundary=20cf3079bd906513ce04d3570e9f X-Virus-Checked: Checked by ClamAV on apache.org --20cf3079bd906513ce04d3570e9f Content-Type: text/plain; charset=ISO-8859-1 Command line running: cd sandbox/prototype/exec/ref mvn exec:java -Dexec.mainClass="org.apache.drill.exec.ref.ReferenceInterpreter" && cat /opt/data/out.json (note, the cat is required because the only current sink is a json writer and the plan currently states to write to this location.) Data input and query files are at: sandbox/prototype/exec/ref/src/test/resources/ >donuts.json is a small sample data file. >simple_plan.json is a logical plan. On Mon, Jan 14, 2013 at 5:07 PM, Michael Hausenblas < michael.hausenblas@gmail.com> wrote: > > > Given that Java6 is public updates EOL next month, I've targeted Java7. > > You probably need to update your JDK. > > Good point. I suppose this should be in big red blinking letters on top of > the installation instructions. > > 'mvn verify' is a success now for me ... in case anyone is on MacOS 10, > I've used the JDK from [1] and works like a charm. > > Cheers, > Michael > > [1] > http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html > > -- > Michael Hausenblas > Ireland, Europe > http://mhausenblas.info/ > > On 14 Jan 2013, at 16:37, Jacques Nadeau wrote: > > > Given that Java6 is public updates EOL next month, I've targeted Java7. > > You probably need to update your JDK. > > > > Jacques > > > > On Mon, Jan 14, 2013 at 4:31 PM, Michael Hausenblas < > > michael.hausenblas@gmail.com> wrote: > > > >> > >>> Be careful, there are many sharp edges. > >>> > >>> :) > >> > >> > >> Alright ;) > >> > >> So here we go: 'mvn validate' works fine for me but then ... > >> > >> $ mvn verify > >> > >> [INFO] prototype-parent .................................. SUCCESS > [7.166s] > >> [INFO] common ............................................ FAILURE > >> [17.775s] > >> [INFO] contrib-parent .................................... SKIPPED > >> [INFO] pom ............................................... SKIPPED > >> [INFO] exec .............................................. SKIPPED > >> [INFO] java-exec ......................................... SKIPPED > >> [INFO] Logical Plan Execution Reference Implementation ... SKIPPED > >> [INFO] planner ........................................... SKIPPED > >> [INFO] sqlparser ......................................... SKIPPED > >> > >> > >> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to > execute > >> goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:compile > >> (default-compile) on project common: Compilation failure > >> Failure executing javac, but could not parse the error: > >> javac: invalid target release: 1.7 > >> Usage: javac > >> use -help for a list of possible options > >> > >> > >> at > >> > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213) > >> at > >> > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) > >> at > >> > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) > >> at > >> > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84) > >> at > >> > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59) > >> at > >> > org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183) > >> at > >> > org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161) > >> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:319) > >> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156) > >> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537) > >> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196) > >> at org.apache.maven.cli.MavenCli.main(MavenCli.java:141) > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> at > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > >> at > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > >> at java.lang.reflect.Method.invoke(Method.java:597) > >> at > >> > org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290) > >> at > >> > org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230) > >> at > >> > org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409) > >> at > >> > org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352) > >> Caused by: org.apache.maven.plugin.CompilationFailureException: > >> Compilation failure > >> Failure executing javac, but could not parse the error: > >> javac: invalid target release: 1.7 > >> Usage: javac > >> use -help for a list of possible options > >> > >> > >> at > >> > org.apache.maven.plugin.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:516) > >> at > >> org.apache.maven.plugin.CompilerMojo.execute(CompilerMojo.java:114) > >> at > >> > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101) > >> at > >> > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209) > >> > >> > >> > >> Cheers, > >> Michael > >> > >> -- > >> Michael Hausenblas > >> Ireland, Europe > >> http://mhausenblas.info/ > >> > >> On 14 Jan 2013, at 16:06, Jacques Nadeau > wrote: > >> > >>> Be careful, there are many sharp edges. > >>> > >>> :) > >>> > >>> On Mon, Jan 14, 2013 at 4:02 PM, Michael Hausenblas < > >>> michael.hausenblas@gmail.com> wrote: > >>> > >>>> > >>>> Cool stuff, Jacques - will give it a shot ASAP! > >>>> > >>>> Cheers, > >>>> Michael > >>>> > >>>> -- > >>>> Michael Hausenblas > >>>> Ireland, Europe > >>>> http://mhausenblas.info/ > >>>> > >>>> On 14 Jan 2013, at 15:56, Jacques Nadeau > >> wrote: > >>>> > >>>>> I've been pulling together a reference logical plan interpreter. I'm > >>>>> working with Ted to get it inside the Drill sandbox. For now, you > can > >>>> find > >>>>> it on my repo at > https://github.com/jacques-n/incubator-drill(prototype > >>>>> branch) > >>>>> > >>>>> > >>>>> > >>>>> The goals of the reference interpreter are: > >>>>> > >>>>> > >>>>> - To provide a simple way to run a Logical Plan against some sample > >>>> data > >>>>> and get back the expected result > >>>>> - Allow work to start on the parsers while we scale up the > performance > >>>>> and capabilities of the execution engine and optimizer. > >>>>> - Allow evaluation work on particular technical approaches such as > >>>>> exploring the impact of hierarchical and schema less data on query > >>>>> evaluation. > >>>>> > >>>>> These goals do not include performance, memory handling, or > >>>>> efficiency. Currently, > >>>>> the interpreter is a single node/thread process. This will change > >>>> shortly > >>>>> so that it also run as a clustered process. > >>>>> > >>>>> The entry point is inside the /sandbox/prototype/exec/ref module: > >>>>> org.apache.drill.exec.ref.ReferenceInterpreter.main(); The example > >>>> program > >>>>> utilizes two resources: simple-plan.json and donuts.json and outputs > >> data > >>>>> to /opt/data/out.json. > >>>>> > >>>>> > >>>>> Some of things that 'work'. > >>>>> > >>>>> > >>>>> - Read/write basic json. > >>>>> - ROPs (reference operators): Filter, Transform, Group, Aggregate > >>>>> (simple), Order, Union. > >>>>> - Example aggregate and basic functions including sum, count, > >> multiply, > >>>>> add, compare, equals. > >>>>> > >>>>> Basic glossary/concepts (we'll get this on the wiki/javadocs): > >>>>> > >>>>> > >>>>> - LOP: Logical Operator. An implementation agnostic data flow > >> operator > >>>>> utilized by the Logical Plan. > >>>>> - ROP: Reference Operator: A reference operator implementation that > >>>>> pairs with a LOP. > >>>>> - FunctionDefinition: A definition of a particular function. > >> Describes > >>>>> a set of aliases, an allowable set of input arguments and an > interface > >>>> that > >>>>> will attempt to determine output type. > >>>>> - BasicEvaluator: An implementation of a particular non-aggregate > >>>>> expression. Receives a record pointer at creation time. Returns a > >>>>> DataValue. > >>>>> - AggregateEvaluator: An implementation of a particular aggregating > >>>>> function. Is provided a record pointer at creation time. Expects > >>>> regular > >>>>> calls to addRecord() followed by a call to eval() which provides the > >>>>> aggregate value. > >>>>> - DataValue: A pointer to a particular data value. Implementation > >>>>> classes includes things like ScalarLong, ScalarBytes, SimpleMapValue > >>>> and > >>>>> SimpleArrayValue. > >>>>> > >>>>> The standard record iterator utilized between each ROP utilizes the > >>>>> org.apache.drill.exec.ref.RecordIterator interface. This is somewhat > >>>>> inspired by the AttributeSource concepts from within the Lucene > >> project. > >>>>> (I'm planning to extend these concepts all the way to the individual > >>>>> DataValues.) > >>>>> > >>>>> > >>>>> > >>>>> My next goals are to add tests, finish adding ROPs, add local and > >> remote > >>>>> exchange nodes (parallelization), add a bunch of documentation and > >>>> extract > >>>>> out the Execution plan as a separate intermediate representation. > >>>>> > >>>>> > >>>>> > >>>>> It needs a lot more evaluators to be a true reference interpreter (as > >>>> well > >>>>> as the rest of the ROPs). The existing ones can be utilized as > >>>> prototypes. > >>>>> Anyone interested in ripping through a bunch of additional evaluators > >> and > >>>>> associated FunctionDefinitions? > >>>> > >>>> > >> > >> > > --20cf3079bd906513ce04d3570e9f--