nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Bende <bbe...@gmail.com>
Subject Re: Logstash/ Filebeat/ Lumberjack -> Nifi
Date Fri, 06 May 2016 13:56:42 GMT
Hi Conrad,

I am not that familiar with LogStash, but as you mentioned there is a PR
for Lumberjack processors [1] which is not yet released, but could help if
you are already using LogStash.
If LogStash has outputs for TCP, UDP, or syslog then like you mentioned, it
seems like this could work well with ListenTCP, ListenUDP, or ListenSyslog.

I think the only additional benefit of Lumberjack is that it is an
application level protocol that provides additional reliability on top of
the networking protocols, meaning if ListenLumberjack receives an event
over TCP it would then acknowledge that NiFi has successfully received and
stored the data, since TCP can only guarantee it was delivered to the
socket, but the application could have dropped it.

Although MiNiFi is not yet released, a possible solution is to run
standalone NiFi instances on the servers where your logs are, with a simple
flow like TailFile -> Remote Process Group which sends the logs back to a
central NiFi instance over Site-To-Site.

Are you able to share any more info about what kind of logs they are and
how they are being produced?
If they are coming from Java applications using logback or log4j, and if
you have control over those applications, you can also use a specific
appender like a UDP appender to send directly over to ListenUDP in NiFi.

Hope that helps.

-Bryan

[1] https://github.com/apache/nifi/pull/290

On Fri, May 6, 2016 at 3:33 AM, Conrad Crampton <conrad.crampton@secdata.com
> wrote:

> Hi,
> Some advice if possible please. Whilst I would love to wait for the MiNiFi
> project realise its objectives as this sounds exactly what I want from the
> initial suggestions I have a pressing need to shift some log files on
> remote servers (to my DC) to my NiFi cluster. Having a quick look at
> LogStash it would look to provide what I want but there doesn’t (yet – I’m
> aware of the work going on Lumberjack processor but not in current release)
> appear to be a simple way of getting files from Logstash to Nifi.
>
> The options currently would appear to be use any number of output plugins
> in Logstash – TCP, UDP, syslog, kafka, http, rabbitmq then use the
> equivalent receiver in Nifi (with some intermediate service in some cases –
> Kafka, rabbitmq).
>
> Can any one suggest the ‘best’ way here? I’m trying to prove a point about
> cutting out some other intermediate product so this is something that has
> to be in production now – I can always refactor at a later date to have a
> ‘better’ solution (MiNiFi ??).
>
> Why don’t I ask on Logstash forums? You folks have always been a great
> help before ;-)
>
> Thanks
> Conrad
>
> Nb. Of course not saying Logstash folks wouldn’t be equally helpful :-)
>
>
> SecureData, combating cyber threats
>
> ------------------------------
>
> The information contained in this message or any of its attachments may be
> privileged and confidential and intended for the exclusive use of the
> intended recipient. If you are not the intended recipient any disclosure,
> reproduction, distribution or other dissemination or use of this
> communications is strictly prohibited. The views expressed in this email
> are those of the individual and not necessarily of SecureData Europe Ltd.
> Any prices quoted are only valid if followed up by a formal written quote.
>
> SecureData Europe Limited. Registered in England & Wales 04365896.
> Registered Address: SecureData House, Hermitage Court, Hermitage Lane,
> Maidstone, Kent, ME16 9NT
>

Mime
View raw message