metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Casey Stella <ceste...@gmail.com>
Subject Re: Metron-265 Model as a Service
Date Mon, 04 Jul 2016 21:35:28 GMT
This is great!  I'll capture any requirements that anyone wants to
contribute and ensure that the proposed architecture accommodates them.  I
think we should focus on a minimal set of requirements and an architecture
that does not preclude a larger set.  I have found that the best driver of
requirements are installed users. :)

For instance, I think a lot of questions about how often to update a model
and such should be represented in the architecture by the ability to
manually update a model, so as long as we have the ability to update,
people can choose when and where to do it (i.e. time based or some other
trigger).  That being said, we don't want to cause too much effort for the
user if we can avoid it with features.

In terms of the questions laid out, here are the constraints from the
proposed architecture as I see them.  It'd be great to get a sense of
whether these constraints are too onerous or where they're not opinionated
enough :

   - Model versioning and retention
   - We do have the ability to update models, but the training and decision
      of when to update the model is left up to the user.  We may want to think
      deeply about when and where automated model updates can fit
      - Also, retention is currently manual.  It might be an easier win to
      set up policies around when to sunset models (after newer versions are
      added, for instance).
   - Model access controls management
   - The architecture proposes no constraints around this.  As it stands
      now, models are held in HDFS, so it would inherit the same security
      capabilities from that (user/group permissions + Ranger, etc)
   - Requirements around concept drift
   - I'd love to hear user requirements around how we could automatically
      address concept drift.  The architecture as it's proposed let's the user
      decide when to update models.
   - Requirements around model output
   - The architecture as it stands just mandates a JSON map input and JSON
      map output, so it's up to the model what they want to pass back.
      - It's also up to the model to document its own output.
   - Any model audit and logging requirements
   - The architecture proposes no constraints around this.  I'd love to see
      community guidance around this.  As it stands, we just log using the same
      mechanism as any YARN application.
   - What model metrics need to be exposed
   - The architecture proposes no constraints around this.  I'd love to see
      community guidance around this.
      - Requirements around failure modes
   - We briefly touch on this in the document, but it is probably not
      complete.  Service endpoint failure will result in blacklisting from a
      storm bolt perspective and node failure should result in a new container
      being started by the Yarn application master.  Beyond that, the
      architecture isn't explicit.


On Mon, Jul 4, 2016 at 1:49 PM, James Sirota <jsirota@apache.org> wrote:

> I left a comment on the JIRA.  I think your design is promising.  One
> other thing I would suggest is for us to crowd source requirements around
> model management.  Specifically:
>
> Model versioning and retention
> Model access controls management
> Requirements around concept drift
> Requirements around model output
> Any model audit and logging requirements
> What model metrics need to be exposed
> Requirements around failure modes
>
> 03.07.2016, 14:00, "Casey Stella" <cestella@gmail.com>:
> > Hi all,
> >
> > I think we are at the point where we should try to tackle Model as a
> > service for Metron. As such, I created a JIRA and proposed an
> architecture
> > for accomplishing this within Metron.
> >
> > My inclination is to be data science language/library agnostic and to
> > provide a general purpose REST infrastructure for managing and serving
> > models trained on historical data captured from Metron. The assumption is
> > that we are within the hadoop ecosystem, so:
> >
> >    - Models stored on HDFS
> >    - REST Model Services resource-managed via Yarn
> >    - REST Model Services discovered via Zookeeper.
> >
> > I would really appreciate community comment on the JIRA (
> > https://issues.apache.org/jira/browse/METRON-265). The proposed
> > architecture is attached as a document to that JIRA.
> >
> > I look forward to feedback!
> >
> > Best,
> >
> > Casey
>
> -------------------
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message