drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charles Givre <cgi...@gmail.com>
Subject Re: [DISCUSS]: Thoughts
Date Thu, 30 Jan 2020 22:11:36 GMT
Thanks everyone for their thoughts, see responses inline. 

> On Jan 30, 2020, at 1:42 PM, Paul Rogers <par0328@yahoo.com.INVALID> wrote:
> Hi All,
> Great discussion. Charles, thanks for starting it. Thanks Isabel for your insights.
> In my mind, question 4 (mission) is the most critical: what users do we want to serve?
What problems should Drill solve for these users? We've touched on this topic in previous
threads, perhaps we need to continue that discussion.
> IMHO* Drill, unfortunately, did not win the "data lake" wars. Cloud vendors won for data
lakes at scale. Snowflake won for corporate data analytics as a service. Presto won for "non-aligned"
data lakes. So, where can Drill add value? This is what folks in Silicon Valley call a "pivot."

I do think Paul's assessment is correct and that Drill should do a pivot as to the mission
of Drill.  From my experience, Drill was never about big data anyway, it was more about being
able to access many different types of data (and platforms) quickly using a common language.
IMHO this is a HUGE industry problem that will only get bigger as more and more types of data
are being generated.  Drill really shines in this regard in that you can literally download
it, unzip it and it works.  No config needed. (Unless you are using Windows ...)

> Again, IMHO, perhaps there is a future in app data integration: the ability to pull,
and combine, data from the myriad S3 file formats and specialized DBs that make up modern
apps. Presto must see this opportunity also: they aregoing all out to dominate this area;
having added many new connectors in the last year.

See above, but this has been my view of Drill from the beginning.  In my line of work (security
& analysis) Drill is an amazing tool that when I show it to security pros they absolutely
love it.  My view here is that we should take the same strategy.  Make Drill able to connect
to as many formats and systems as possible.  This also means API improvements such that it
is easier and easier to build storage and format plugins so that more people can do it. 

> Perhaps Ted, with his wide experience talking to users and the open source community,
can offer us suggestions.
> Realistically, defining a mission must be done in conjunction with point 2: community
building. If we reach out the community and pay close attention to where we find interest,
that will help identify an un- (or under-) served niche where we add unique value. (This is,
in fact, basic marketing.)

Concur... We really could use some assistance here.  From the hangout, I really agreed with
Vadim's sentiment about Drill marketing.  When I present Drill, the aspirational statement
that I use is:

Drill is a universal translator for data.  It can query any kind of data*, no matter where
it is*, or how big it is**. 

* Well... almost
**. Within reason

> Once we determine our target "market", we'll have a better sense where to drive the project,
documentation, user experience and developer experience to satisfy that market. Thanks to
the many outstanding people who contributed to the project over the years, Drill is actually
pretty good compared to many projects, Having a mission will tell us where we need to improve

Everything I've seen indicates to me that there is a market for this kind of capability. 

> So, how do we proceed?
> Thanks,
> - Paul
> * In my *humble* opinion. I've read that many folks now read the "h" as "honest."
>    On Thursday, January 30, 2020, 8:46:01 AM PST, Charles Givre <cgivre@gmail.com>
> I'd agree with Ted's thoughts.  In general, my thought was that we need to work on getting
more users.  More users likely will lead to more contributors, so the goal being to make their
experience as good as possible such that more and more people will want to use Drill and tell
their friends.
> Also, we do need to work on the developer documentation and in general making it easier
to develop for Drill. Paul and Volodmyr have been doing great work in cleaning up code and
providing good docs for it.  Again, if someone wants want to develop for Drill and finds the
code to be extremely difficult to understand, they'll give up and do something else.
> The project that Isabel mentioned as an example was the original HTTPD project, which
lacks corporate backing and yet is thriving.  Anyway, I do think we should have a continued
discussion about the vision and/or mission of Drill.
> -- C
>> On Jan 30, 2020, at 11:18 AM, Ted Dunning <ted.dunning@gmail.com> wrote:
>> Igor,
>> Good documentation and first 5-minute experience are very important, but
>> not because a long-term contributor will see it and commit their spare time
>> for the next five years on that basis. It is more about preventing early
>> attrition of contributors who might find the project very exciting due to
>> silly factors. That can easily happen if the documentation is bad because
>> it increases the frustration a potential contributor feels early on. If
>> they can't try the software and get something interesting, then we are
>> likely to lose the battle for attention span.
>> And frankly, it isn't just the developer that we need to attract and
>> retain. A user who never contributes a line of code is part of the
>> community and can easily be a net positive if they only report problems and
>> occasionally tell people what they are doing.
>> On Thu, Jan 30, 2020 at 7:00 AM Igor Guzenko <ihor.huzenko.igs@gmail.com>
>> wrote:
>>> Hello Charles,
>>> Thank you very much for starting this important discussion. These are all
>>> important things but at the current moment, I don't have a clear vision
>>> where we could start from. The item which is most interesting to me is the
>>> second one, but I've never been involved in building an open-source
>>> community, don't even know where to start. I'm not sure that just making
>>> good documentation and the first impression will attract developers with
>>> strong motivation to contribute.  So I'm very excited to learn about
>>> projects which managed to build such a community, maybe we really could
>>> find some new fresh ideas about how to attract new community members.
>>> Thanks,
>>> Igor
>>> On Thu, Jan 30, 2020 at 4:18 PM Charles Givre <cgivre@gmail.com> wrote:
>>>> Hello all,
>>>> I mentioned in the Drill hangout last week that I had spoken with one of
>>>> the original mentors for the Drill project (Isabel Drost-Fromm) and asked
>>>> her advice about the future of Drill.  To paraphrase what she told me:
>>>> 1.  There are two ways for open source projects to succeed.  The first
>>>> and riskier approach is with a single corporate sponsor.  The obvious
>>> risks
>>>> are that since the corporate sponsor is footing the bill, they will
>>>> prioritize their own needs over and sometimes against community needs.
>>>> (This is not unique to Drill). The slower but less risky approach is to
>>>> build a community around a project, join forces and slowly drive it
>>>> forward.  She pointed out that some of the Apache foundation's longest
>>>> running projects were run in this way.
>>>> 2.  We should focus our efforts on community building:  She suggested a
>>>> lot of what she described as "would be obvious in retrospect" such as
>>>> making sure the documentation is really solid, and having a user
>>> experience
>>>> in the beginning.  She said we should use the resources of the Apache
>>>> foundation to help publicize new releases etc.  Also we should make it
>>> easy
>>>> to become a committer.  IMHO, I would add that we really should seek out
>>>> additional code reviewers as we don't have enough and PRs take a long
>>> time
>>>> to get approved.
>>>> 3.  Do a lot of what a vendor would do:  Update the website and
>>>> documentation to reflect things like: who is using Drill, who is offering
>>>> professional support for Drill etc.
>>>> 4.  Define a mission:  We should work to define a mission for Drill?  IE
>>>> Why does/should it exist and what business problem does it solve?  IMHO
>>> it
>>>> solves a very large one, but more people need to know about it.  That's
>>> why
>>>> I'm not giving up on it yet.
>>>> @Isabel, I hope I captured the essence of what you were telling me here.
>>>> Thanks everyone,
>>>> --C

View raw message