spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörn Franke <jornfra...@gmail.com>
Subject Re: Spark for offline log processing/querying
Date Mon, 23 May 2016 06:08:59 GMT
Do you want to replace ELK by Spark? Depending on your queries you could do as you proposed.
However, many of the text analytics queries will probably be much faster on ELK. If your queries
are more interactive and not about batch processing then it does not make so much sense. I
am not sure why you plan to use Presto.

> On 23 May 2016, at 07:28, Mat Schaffer <mat@schaffer.me> wrote:
> 
> I'm curious about trying to use spark as a cheap/slow ELK (ElasticSearch,Logstash,Kibana)
system. Thinking something like:
> 
> - instances rotate local logs
> - copy rotated logs to s3 (s3://logs/region/grouping/instance/service/*.logs)
> - spark to convert from raw text logs to parquet
> - maybe presto to query the parquet?
> 
> I'm still new on Spark though, so thought I'd ask if anyone was familiar with this sort
of thing and if there are maybe some articles or documents I should be looking at in order
to learn how to build such a thing. Or if such a thing even made sense.
> 
> Thanks in advance, and apologies if this has already been asked and I missed it!
> 
> -Mat
> 
> matschaffer.com

Mime
View raw message