spot-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raymundo Panduro <rpand...@apache.org>
Subject Re: Spot Suspicious Connects Description and questions related to 'feedback' from UI to ML
Date Thu, 25 May 2017 18:12:45 GMT
Yes, now is working! Thx

On Thu, May 25, 2017 at 1:11 PM, Edwards, Brandon <brandon.edwards@intel.com
> wrote:

> Ok this should work:
> https://www.dropbox.com/s/qezo46152u65tpu/suspiciousConnectsDescription_
> 5_22_2017.pdf?dl=0
>
> Can someone confirm? THX
>
> On 5/25/17, 11:08 AM, "Segerlind, Nathan L" <nathan.l.segerlind@intel.com>
> wrote:
>
>     The link does not work for me.... I keep getting a message that reads
> "folder Spot does not exist"
>
>     -----Original Message-----
>     From: Edwards, Brandon [mailto:brandon.edwards@intel.com]
>     Sent: Thursday, May 25, 2017 10:57 AM
>     To: dev@spot.incubator.apache.org
>     Subject: Re: Spot Suspicious Connects Description and questions
> related to 'feedback' from UI to ML
>
>     Oh I had the scoring values reversed. Thanks Alan!
>
>     Also here is a link to the file on Dropbox:
>     https://www.dropbox.com/home/Spot?preview=
> suspiciousConnectsDescription_5_22_2017.pdf
>
>     Brandon
>
>     On 5/25/17, 10:50 AM, "Alan Ross" <alan@apache.org> wrote:
>
>         On the scoring piece.  1 has traditionally been "Bad" and 3 has
> been
>         "Benign".  Are we changing that?
>
>         Alan
>
>         On Thu, May 25, 2017 at 10:49 AM, Alan Ross <alan@apache.org>
> wrote:
>
>         > I don't believe this list permits attachments Brandon.  Perhaps
> post it to
>         > google docs and send out a link?
>         >
>         > Alan
>         >
>         > On Thu, May 25, 2017 at 10:27 AM, Edwards, Brandon <
>         > brandon.edwards@intel.com> wrote:
>         >
>         >> Hi all,
>         >>
>         >>
>         >>
>         >> I am attaching the document that describes how Spot uses LDA in
> order to
>         >> perform anomaly detection on network events. I have also
> received multiple
>         >> questions related to how the ‘user scoring’ (‘feedback’) of
> particular
>         >> items in the suspicious connects report (in the UI layer) is
> used in ML. We
>         >> have not provided much detail on this functionality in the
> attached
>         >> document. I thought I’d put an explanation out there and we can
> discuss
>         >> questions related to my explanation and discuss what additional
> info should
>         >> be included in the attached document.
>         >>
>         >>
>         >>
>         >> The Spot team feels that changes are needed to this ‘feedback’
>         >> functionality, and see these changes as happening concurrent
> with
>         >> improvements to the ability for context from an LDA model
> trained on a
>         >> given batch of data to be carried forward to the next training
> run (or even
>         >> training in a streaming use case). The value of ‘feedback’ is
> dependent on
>         >> the quality of the model-context we can carry over.
>         >>
>         >>
>         >>
>         >> The idea for feedback is as follows. The items that are scored
> with a 1
>         >> (i.e. the user identifies the item as benign and so does not
> want to see it
>         >> in the suspicious connects report anymore) will be used for
> letting the
>         >> machine learning component know that such an entry should not
> be considered
>         >> as suspicious anymore. Currently this is done by injecting
> artificial log
>         >> entries into the next batch of data so that LDA sees many such
> entries and
>         >> therefore no longer sees them as anomalies.
>         >>
>         >>
>         >>
>         >> We have ideas for other ways to allow this functionality - for
> example we
>         >> could filter entries matching the identified pattern from the
> next batch
>         >> run BEFORE ML runs on the batch. For items that are scored by
> the user in
>         >> the UI as ‘3’ (for example the user sees an ip as so suspicious
> that we
>         >> want to see all future log entries associated to that ip) we
> could filter
>         >> future items matching such a pattern in order to skip ML and
> instead report
>         >> them in a separate pane of the UI or insert them to the top of
> the most
>         >> suspicious events.
>         >>
>         >>
>         >>
>         >> Comments, Questions?
>         >>
>         >> Brandon
>         >>
>         >
>         >
>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message