drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Omernik <j...@omernik.com>
Subject Re: deploy dockerized drill cluster
Date Mon, 01 Aug 2016 12:05:31 GMT
So I think part of the problem here is the options on json are stored in
the zookeeper. Are you running the Zookeepers in the docker containers, in
separate docker containers, or completely separate?  For those settings,
I'd boot strap the cluster by itself, get it up and running, and then use
the ALTER SYSTEM set  command, either from SQLLINE or using the Rest API.
Then they will persist int he Zookeeper information.

I know I asked a similar question, because I wanted new clusters to be able
to be setup to use specified admin users. (I.e. I wanted to provide my list
of users that were admins prior to the first drill bit starting, and there
wasn't an easy way to do that at the time I asked).



On Sun, Jul 31, 2016 at 8:07 PM, Jesse Yates <jesse.k.yates@gmail.com>
wrote:

> Ran into this myself today, so gonna try and close the loop.
>
> Looking at the 1.6 codeline (assuming its the same in 1.7), on startup the
> SystemOptionManager on pulls out the sys.options line from the config and
> uses that to build the PersistentStoreConfig (which get initialized from
> the store in #init()). However, this obviously doesn't take into account
> the config level options, only the startup options since those get pulled
> out to create everything else.
>
> It seems simple enough to wrap the SystemOptionManager into a
> FallbackOptionManager that falls back to the config file, if its not in the
> persistent store (or something like that).
>
> Unless someone already filed a JIRA on this?
>
> On Mon, Jul 25, 2016 at 10:05 AM Scott Kinney <scott.kinney@stem.com>
> wrote:
>
> > it's still not picking up the store.json* config changes
> >
> > The only way I can see to set these is with running ALTER SYSTEM query
> > after drill api is up.
> >
> >
> > ________________________________
> > Scott Kinney | DevOps
> > stem   |   m  510.282.1299
> > 100 Rollins Road, Millbrae, California 94030
> >
> > This e-mail and/or any attachments contain Stem, Inc. confidential and
> > proprietary information and material for the sole use of the intended
> > recipient(s). Any review, use or distribution that has not been expressly
> > authorized by Stem, Inc. is strictly prohibited. If you are not the
> > intended recipient, please contact the sender and delete all copies.
> Thank
> > you.
> >
> > ________________________________________
> > From: John Omernik <john@omernik.com>
> > Sent: Monday, July 25, 2016 8:21 AM
> > To: user
> > Subject: Re: deploy dockerized drill cluster
> >
> > Try (for the sake of the conversation here) using host networking, and
> see
> > if it changes how successful your setup is.  (I know bridged is
> preferred,
> > but try the host side and see what happens)
> >
> > John
> >
> > On Mon, Jul 25, 2016 at 10:06 AM, Scott Kinney <scott.kinney@stem.com>
> > wrote:
> >
> > > I'm running the docker in bridged network mode.
> > >
> > >
> > > ________________________________
> > > Scott Kinney | DevOps
> > > stem   |   m  510.282.1299
> > > 100 Rollins Road, Millbrae, California 94030
> > >
> > > This e-mail and/or any attachments contain Stem, Inc. confidential and
> > > proprietary information and material for the sole use of the intended
> > > recipient(s). Any review, use or distribution that has not been
> expressly
> > > authorized by Stem, Inc. is strictly prohibited. If you are not the
> > > intended recipient, please contact the sender and delete all copies.
> > Thank
> > > you.
> > >
> > > ________________________________________
> > > From: John Omernik <john@omernik.com>
> > > Sent: Sunday, July 24, 2016 8:28 AM
> > > To: user
> > > Subject: Re: deploy dockerized drill cluster
> > >
> > > Are you running Drill in host networking or bridged networking?
> > >
> > > On Sat, Jul 23, 2016 at 1:21 PM, Scott Kinney <scott.kinney@stem.com>
> > > wrote:
> > >
> > > > Hm, i must have set those another way in embeded mode. I can't see
> > where.
> > > > Those settings persist between drill restarts.
> > > >
> > > >
> > > >
> > > > ________________________________
> > > > Scott Kinney | DevOps
> > > > stem   |   m  510.282.1299
> > > > 100 Rollins Road, Millbrae, California 94030
> > > >
> > > > This e-mail and/or any attachments contain Stem, Inc. confidential
> and
> > > > proprietary information and material for the sole use of the intended
> > > > recipient(s). Any review, use or distribution that has not been
> > expressly
> > > > authorized by Stem, Inc. is strictly prohibited. If you are not the
> > > > intended recipient, please contact the sender and delete all copies.
> > > Thank
> > > > you.
> > > >
> > > > ________________________________________
> > > > From: Abhishek Girish <abhishek.girish@gmail.com>
> > > > Sent: Friday, July 22, 2016 1:57 PM
> > > > To: Drill User List
> > > > Subject: Re: deploy dockerized drill cluster
> > > >
> > > > You can set boot level start-up options in drill-override.conf [1].
> > But I
> > > > don't think we can do the same with the system options. Someone else
> > can
> > > > comment if there is a workaround.
> > > >
> > > > On why it works for you with drill-embedded, is something I'm trying
> to
> > > > understand. I attempted this and couldn't manage to get those options
> > to
> > > > show up in embedded mode.
> > > >
> > > > [1] https://drill.apache.org/docs/start-up-options/
> > > >
> > > > On Fri, Jul 22, 2016 at 1:20 PM, Scott Kinney <scott.kinney@stem.com
> >
> > > > wrote:
> > > >
> > > > > I have built a drill docker images very much like
> > > > > https://github.com/bigstepinc/apache-drill/blob/master/Dockerfile
> > > > >
> > > > >
> > > > > I volume mount a drill-override.conf file that looks like:
> > > > >
> > > > > drill.exec:{
> > > > >   cluster-id: drill1,
> > > > >   zk.connect: 192.1.1.1:2181",
> > > > >   sys.store.provider.local.path: "/drill-storage",
> > > > >   store.json.all_text_mode: True,
> > > > >   store.json.read_numbers_as_double: True
> > > > > }
> > > > >
> > > > > it seems to be connecting to the zookeeper (otherwise it would fail
> > > > right?
> > > > > i should actually confirm this).
> > > > >
> > > > > I know it is picking up my s3 plugin that i volume mount inside
> > > > > /drill-storage but it's not setting the json all_text_mode and
> > > > > read_numbers_as_double.
> > > > >
> > > > > I have a drill-embeded instance that seems to pick these json
> setting
> > > > from
> > > > > the drill-override file just fine.
> > > > >
> > > > > Can you see what I'm doing wrong?
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ________________________________
> > > > > Scott Kinney | DevOps
> > > > > stem <http://www.stem.com/>   |   m  510.282.1299
> > > > > 100 Rollins Road, Millbrae, California 94030
> > > > >
> > > > > This e-mail and/or any attachments contain Stem, Inc. confidential
> > and
> > > > > proprietary information and material for the sole use of the
> intended
> > > > > recipient(s). Any review, use or distribution that has not been
> > > expressly
> > > > > authorized by Stem, Inc. is strictly prohibited. If you are not the
> > > > > intended recipient, please contact the sender and delete all
> copies.
> > > > Thank
> > > > > you.
> > > > >
> > > >
> > >
> >
> --
> Jesse Yates
> Founder/CEO Fineo.io
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message