spot-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nate Smith <natedogs...@gmail.com>
Subject Re: DNS ingestion
Date Tue, 27 Feb 2018 17:04:17 GMT
Hello,

This email seemed to have slipped through the cracks. 
How is development going?
Is this still something you are interested in contributing?

- nathanael 

> On Sep 15, 2017, at 5:17 AM, Salvatore Elio <Elio.Salvatore@sia.eu> wrote:
> 
> ​Hello,
> 
> 
> 
> for an internal project we have developed a different ingestion process for DNS in order
to have a real time ingestion and support early enrichment of the ingested data.
> 
> The ingestion process is splitted into 2 processes:
> 
> 
> 
> 1)     ​From DNS data to Kafka -  An Akka Streams job based on Pcap4j (https://<https://github.com/kaitoy/pcap4j)>github.com/kaitoy/pcap4j<https://github.com/kaitoy/pcap4j)>)
which​:
> 
> a.     loop through all the filtered UDP packets on port 53 using Pcap4j;
> 
> b.     convert Pcap4j packet objects to AVRO using Twitter Bijection;
> 
> c.      send Avro objects to Kafka.
> 
> 
> 
> 2) From Kafka to hive - a Spark Streaming job that read new messages on Kafka and write
them in partitioned HDFS parquet folder readable by Hive/Impala.
> ​
> 
> ​We would like to know your thoughts about this and if this could be integrated into
apache spot. If it is of interest we can share the code.
> 
> 
> 
> 
> 
> Thanks
> 
> [SIA logo]
> ________________________________
> 
> *******************Internet Email Confidentiality Footer*******************
> Qualsiasi utilizzo non autorizzato del presente messaggio nonché dei suoi allegati è
vietato e potrebbe costituire reato. Se ha ricevuto per errore il presente messaggio, Le saremmo
grati se ci inviasse, via e-mail, una comunicazione al riguardo e provvedesse nel contempo
alla distruzione del messaggio stesso e dei suoi eventuali allegati. Le dichiarazioni contenute
nel presente messaggio nonche' nei suoi eventuali allegati devono essere attribuite al mittente
e non possono essere necessariamente considerate come autorizzate da SIA S.p.A.; le medesime
dichiarazioni non impegnano SIA S.p.A. nei confronti del destinatario o di terzi. SIA S.p.A.
non si assume alcuna responsabilita' per eventuali intercettazioni, modifiche o danneggiamenti
del presente messaggio e-mail.
> 
> Any unauthorized use of this e-mail or any of its attachments is prohibited and could
constitute an offence. If you are not the intended addressee please advise immediately the
sender by using the reply facility in your e-mail software and destroy the message and its
attachments. The statements and opinions expressed in this e-mail message are those of the
author of the message and do not necessarily represent those of SIA S.p.A. Besides, The contents
of this message shall be understood as neither given nor endorsed by SIA S.p.A.. SIA S.p.A.
does not accept liability for corruption, interception or amendment, if any, or the consequences
thereof.

Mime
View raw message