lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jack Krupansky" <j...@basetechnology.com>
Subject Re: Parquet dictionary encoding & bit packing
Date Sun, 15 Sep 2013 21:50:50 GMT
Okay, but what exactly does Parquet have to offer to a search engine?

I mean, is it simply an alternate form of codec?

Would it merely reduce I/O and mass storage requirements?

Would it impact "search" performance at all?

Would it add a significant search start-up "warming" overhead? Or, does it 
offer some magic that would in fact dramatically reduce the time to do the 
first query?

Or, is it merely an alternative format for ingestion of an input stream? 
Like, say, better than JavaBin? Or, maybe for more efficient internode 
transfers of documents for SolrCloud?

-- Jack Krupansky

-----Original Message----- 
From: Otis Gospodnetic
Sent: Sunday, September 15, 2013 5:17 PM
To: dev@lucene.apache.org
Subject: Parquet dictionary encoding & bit packing

Hi,

I was reading the Parquet announcement from July:
https://blog.twitter.com/2013/announcing-parquet-10-columnar-storage-for-hadoop

And a few things caught my attention - Dictionary encoding and
(dynamic) bit packing.  This smells like something Adrien likes to eat
for breakfast.

Over in the Hadoop ecosystem Parquet interest has picked up:
http://search-hadoop.com/?q=parquet

I thought I'd point it out as I haven't seen anyone bring this up.  I
imagine there are ideas to be borrowed there.

Otis
--
Solr & ElasticSearch Support -- http://sematext.com/
Performance Monitoring -- http://sematext.com/spm

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message