drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John Omernik (JIRA)" <j...@apache.org>
Subject [jira] [Created] (DRILL-4052) Allow variables in drill-override.conf
Date Mon, 09 Nov 2015 14:26:11 GMT
John Omernik created DRILL-4052:

             Summary: Allow variables in drill-override.conf
                 Key: DRILL-4052
                 URL: https://issues.apache.org/jira/browse/DRILL-4052
             Project: Apache Drill
          Issue Type: Improvement
          Components: Tools, Build & Test
    Affects Versions: 1.1.0
         Environment: All
            Reporter: John Omernik
             Fix For: Future

Basic variables, starting with "$HOSTNAME" if that simplified things would be a huge help
to managing clusters of drill bits, especially in multi-tenant environments.  

(Note: Perhaps being able to use all ENV variables through a $ENV:ENVVARNAME would address
the issue completely, giving ultimate flexibility). 

The use case/story I have is a drill cluster where I am running the individual drill bits
in Marathon on Mesos. Since I am using MapR-FS (although this could work without MapR through
Marathon downloading the config at run time) I point all drill bits to one config location.
  This allows me to make a change in one place, making for easy changes and configuration

For the drill-env.sh, since that is shell script, I found I could easily put in items like
specifying log locations that allowed me to use one shared log location for all drillbits,
and then specifying that each logfile had the hostname of the drillbit, so I could keep all
logs together and thus troubleshoot much easier.  

When it came to drill-override.conf, I wanted to specify spill directory locations. Specifically,
I wanted to use local volumes in MapR-FS.  It allows me to use a clustered filesystem, but
have that data only exit on the node that the drill bit is running on.  So, Scheme: maprfs
works, but what to specify for the location? I didn't want to specify the same location, /tmp/spill
as then it would be shared by drillbits, and not guaranteed to be local to the drill bit (giving
up on performance).  I wanted to create a volume that could be mounted /tmp/$hostname/drillspill
  I could create the volume in drill-env.sh, but I could not specify that in the drill-override
without creating a separate override file for each node. 

Given scaling and multiple users/clusters, I really want to avoid that situation (a config
for each drill bit), hence creating this JIRA for the ability to use variables in the drill-override.
Happy to discuss other use cases as well. 

This message was sent by Atlassian JIRA

View raw message