drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Altekruse <altekruseja...@gmail.com>
Subject meeting notes 10/22/13
Date Tue, 22 Oct 2013 16:49:36 GMT
Hello All,

Here are the notes from todays hangout. Michael, can you copy them into the
google doc?

participants: Jacques, Micheal hausenblas, Lisen Mu, Yash Sharma, Jinfeng,
Jason Altekruse, Harri, Steven Phillips, Timothy Chen, Julien Hyde

New employee at MapR: Jinfeng
    - couple more in the next month

    - merged limit
    - clarify VVs
        - never access internal state of VV when it is invalid
    - release notes

    - ordered partitioner
        - abstract out distributed cache interface
    - continue to work on spooling to disk
        - look at sort and ordered hash partitioner

    - name of functions
        - separate class for operators and functions for more clarity
            - different operators have their own class files

    - fork of Drill
        - data pushed form leaves rather than pulled from root
        - we have been thinking about this same problem
            - don't want to wait for IO all the time
            - pre-fetch rather than push
            - in a join you might get pushed a huge amount of data when you
aren't ready for it
            - stream processing
                - alternative concept around foreman
                - not quite right for streams
                - resource allocation
                    - not as much for resource requirements
            - space saving
            - acceptable - not precise
        - data assembly - business logic
            - approximations will be important to drill
            - no serious thinking about sampling
            - certain types of scanners should support sampling
                - hard with some without reading all data anyway
                - Hbase might be easier to do a scan
            - doing it with their own business logic and statistics
                - hard to generalize

    - not much for updates
    - pick up with amazon ec2 docs
        - had problem where we need 8 gigs
        - cannot get it running on free micro instance
        - got it working removing the direct memory flag in POM
        - tim - out of memory exception right away
            - was this with or without changing the option for direct

    - wir patch in
    - amp labs big data benchmark
        - having numbers for performance evaluation
        - set up on their repo for drill datasets
        - installing HDFS to all of the nodes
        - doesn't look to complicated
    - cannot submit sql in distributed mode because of bad optimizer
    - recent review board patches
        - describe code more completely
        - hard to review without docs
        - Julien - single powerpoint slide per operator
        - google doc? like the logical plan doc

    - code gen portion of merging receiver
    - no blockers
        - getting to code review soon

    - joined hortonworks
    - working on optiq
    - helping hive, but also working on Drill
    - making optiq everything it can be
    - splitting JDBC into thin client
        - thinking about it, no implementation yet
        - right now pushing sorts down to Mongo
    - jacques - session next week on JDBC?
    - roadmap on optiq
        - commit logs tell some of the story
        - roadmap would be helpful
        - will put out call for optiq users like drill
        - put together feature list for next release(s)
        - next 6 months, want to be agile, but wants to be more predictable
        - Jinfeng will be working with optimizer and optiq

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message