drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alla Gorina <agor...@fairwaytech.com>
Subject how to use Drill as the OLAP tool in the data warehouse reporting app
Date Tue, 06 Dec 2016 17:06:03 GMT
Dear Drill Community,

I am working of evaluating Drill as an OLAP query engine for a new open source product in
the education domain.  Just to give you some background: the solution we are building is a
reporting tool for the student assessments outcomes. When a student takes an assessment test,
it gets scored and then the outcome of the scoring along with the assessment details is loaded
into the reporting warehouse.  We are building that warehouse and the reporting tool on top
of it. As part of the reporting tool we need to report on individual tests, as well as the
aggregated data. The solution has to have an OLTP data store for the individual test reporting,
and we are considering Drill for the analytical reports. As you could see our data is pretty
well structured and the volume is not ‘big’ data, but is estimates at about 150+ million
assessments outcomes.  I see that Drill performs the best with Parquet file store. And I have
two questions:
1.     What is the best way to create Parquet files as the data flow in. Please consider that
we need to support data update and deletes. Also the volume of data flow may vary. I see that
some people capture data for a day and then convert them into Parquet. Is this the way to
go?  I am concerned that:
a.     we may end up with a lot of small size Parquet files
b.     for delete/updates we will need to delete the existing files with the changed records
and recreate them. How to handle the system being up while we are doing this change?
2.     Would it be better for us instead of creating Parquet files, point Drill to our OLTP
data store? We know that OLTP data store by itself does not perform the queries we need within
the desired speed (under 10 sec.). Would Drill be able to imrpove performance in this case?

Thank you in advance for your help!
Alla Gorina

Principal Developer

Fairway Technologies Inc. | Our People, Your Solution.®

[http://www.fairwaytech.com/media/cell-icon.gif]  (858) 774-1067(mobile)

fairwaytech.com<http://www.fairwaytech.com/> | [http://www.fairwaytech.com/wp-content/uploads/2015/11/linkedin_email.gif]
<http://www.linkedin.com/company/fairway-technologies>  [http://www.fairwaytech.com/wp-content/uploads/2015/11/twitter_email.gif]
<http://twitter.com/FairwayTech>  [http://fairwaytech.com/wp-content/uploads/2015/11/facebook_email.gif]
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message