lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Courtade <courtadej...@gmail.com>
Subject Re: solr audit logging
Date Wed, 07 Dec 2016 02:21:59 GMT
They wanted out of the box solutions.

This is what I found too that it would be custom. i was hoping i just was
not finding something obvious.


Jeff Courtade
M: 240.507.6116

On Dec 6, 2016 7:07 PM, "John Bickerstaff" <john@johnbickerstaff.com> wrote:

> You know - if I had to build this, I would consider slurping up the
> relevant log entries (if they exist) and feeding them to Kafka - then your
> people who want to analyze what happened can get those entries again and
> again (Think of Kafka kind of like a persistent messaging store that can
> store log entries or anything you want...)
>
> Of course, how much work you'd have to put into that depends on the
> technical skill of whoever is going to consume this stuff...
>
> Also, a plain old relational database can easily hold these things as well
> -and the code to parse the log messages into some simple tables wouldn't be
> that difficult...  There are probably existing examples / projects...
>
> Naturally - if the standard log entries do NOT get you what you need, then
> it gets to be more of an effort, although adding an extension to Solr isn't
> too hard once you understand the process...
>
> Ping back and let us know what you find in the logs and if you want more
> "advice" -- which you should always take with a grain of salt...
>
> On Tue, Dec 6, 2016 at 3:56 PM, John Bickerstaff <john@johnbickerstaff.com
> >
> wrote:
>
> > If you can identify currently-logged messages that give you what you need
> > (even if you have to modify or process them afterwards) you can easily
> make
> > a custom log4j config that grabs ONLY what you want and dumps it into a
> > separate file...
> >
> > I'm pretty sure I've seen all the request coming through in my SOLR log
> > files...
> >
> > In case that helps...
> >
> > On Tue, Dec 6, 2016 at 2:08 PM, Alexandre Rafalovitch <
> arafalov@gmail.com>
> > wrote:
> >
> >> There is also Jetty level access log which shows the requests, though
> >> it may not show the HTTP PUT bodies.
> >>
> >> Finally, various online monitoring services probably have agents that
> >> integrate with Solr to show what's happening. Usually costs money
> >> though.
> >>
> >> Regards,
> >>     Alex.
> >> ----
> >> http://www.solr-start.com/ - Resources for Solr users, new and
> >> experienced
> >>
> >>
> >> On 6 December 2016 at 14:34, Jeff Courtade <courtadejeff@gmail.com>
> >> wrote:
> >> > Thanks very much the trace idea is a brilliant way to dig into it. Did
> >> not
> >> > occur to me.
> >> >
> >> > I had another coworker suggest the custom
> >> >
> >> > http://lucene.apache.org/solr/6_3_0/solr-core/org/apache/sol
> >> r/update/processor/LogUpdateProcessorFactory.html
> >> >
> >> >
> >> > this is beyond my litmited abilites.
> >> >
> >> >
> >> > I will see what we can dig up out of the logs...
> >> >
> >> >
> >> > the original request was this...
> >> >
> >> >
> >> > "Is there any configuration, plugin, or application that will create
> an
> >> > audit trail for Solr requests? We have teams that would like to be
> able
> >> to
> >> > pull back changes/requests to documents in solr given a time period.
> The
> >> > information they would like to retrieve is the request to solr, where
> it
> >> > came from, and what the request did."
> >> >
> >> >
> >> > I am starting to think there is not a simple solution to this. I was
> >> hoping
> >> > there was an UpdateAudit class or something I could flip a switch on
> or
> >> > some such...
> >> >
> >> >
> >> >
> >> > On Tue, Dec 6, 2016 at 2:20 PM, Alexandre Rafalovitch <
> >> arafalov@gmail.com>
> >> > wrote:
> >> >
> >> >> You could turn the trace mode for everything in the Admin UI (under
> >> >> logs/levels) and see if any of the existing information is sufficient
> >> >> for your needs. If yes, then you change log level in the
> configuration
> >> >> just for that class/element.
> >> >>
> >> >> Alternatively, you could do a custom UpdateRequestProcessor in the
> >> >> request handler(s) that deal with update. Or perhaps
> >> >> LogUpdateProcessor (that's in every standard chain) is sufficient:
> >> >> http://www.solr-start.com/javadoc/solr-lucene/org/
> >> >> apache/solr/update/processor/LogUpdateProcessorFactory.html
> >> >>
> >> >> But it is also possible that the audit.log is something that has a
> >> >> specific format that other tools use. So, you could start from asking
> >> >> how that file would be used and then working backwards into Solr.
> >> >> Which would most likely be a custom URP, as I mentioned earlier.
> >> >>
> >> >> Regards,
> >> >>    Alex.
> >> >> P.s. Remember that there are full document updates and partial
> >> >> updates. What you want to log about that is your business level
> >> >> decision.
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Thanks,
> >> >
> >> > Jeff Courtade
> >> > M: 240.507.6116
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message