drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Omernik <j...@omernik.com>
Subject Drill and Elasticsearch
Date Tue, 21 Feb 2017 21:32:54 GMT
I know there are conversations about an Elasticsearch plugin, however, I
had a recent need to take some data that was accessible in Drill (stored as
a Parquet table) and move it into an Elasticsearch index.  There are about
1 million rows in the source data.

I was learning the technologies here, and so I thought I could use the Rest
API in Drill and then push the data into Elasticsearch.

I found that in using drill, I couldn't run a single query and then batch
them into Elastic Search. To many returned, timeouts etc. So I used LIMIT
1000 and offset.  This allowed the ES side of things to work well, but It
required quite a few of the same query in Drill wasting time and resources.

I guess, I am just looking for ideas, how would YOU get data from Parquet
files into Elastic Search? I have Drill and Spark at the ready, but want to
be able to handle it as efficiently as possible.  Ideally, if we had a well
written ES plugin, I could write a query that inserted into an index and
streamed stuff in... but barring that, what other methods have people used?

Thanks

John

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message