metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Merriman <>
Subject [DISCUSS] Search in REST
Date Mon, 10 Jul 2017 19:14:43 GMT
This discussion is an attempt to clarify some questions and discuss design
decisions related to METRON-1022.

The primary purpose of METRON-1022 is to provide a foundation for building
Metron-specific Elasticsearch (or other search engine implementations)
functions in our REST application.  This translates into 3 features
provided by METRON-1022:  a common approach to setting up a
TransportClient, a search abstraction layer and a simple Elasticsearch
implementation consisting of a single function.  I believe the setup part
is fairly straightforward and doesn't require a detailed discussion.
Please chime in if I'm wrong there.

The first order of business is to all agree on an architectural approach.
How and where should we query Elasticsearch?  METRON-1022 duplicates some
functionality in METRON-990 but is architecturally different.  Instead of
the client-side code interacting directly with Elasticsearch through it's
REST api, this PR interacts with Elasticsearch through the Java api in a
Metron REST service.  I believe there are a couple of advantages to doing
it this way:

   - A Metron-specific search service in REST can be reused by other UIs
   and clients.  It would be possible to make an angular service reusable but
   that would take some work and it would only be reusable in javascript as an
   imported library and not as flexible as a service available over http.
   - Metron provides an integration testing framework for Java-based
   classes.  METRON-1022 leverages this without much additional effort.  It
   would take a lot more work to enable javascript modules to use this.
   - In my experience, the Metron community is much more comfortable with
   developing and reviewing features written in Java as opposed to
   javascript.  I think that is important for foundational pieces like this.

Some arguments to consider for keeping Elasticsearch functions in a
javascript service:

   - More efficient since there is no proxy in the middle (Metron REST
   being the proxy)
   - Eliminates the task of resolving version conflicts that comes with
   adding the Elasticsearch dependency to a Maven module although there are
   ways to make this easier

The second topic to discuss is the search abstraction.  This has been
requested several times and I think there is consensus that we need it.
METRON-1022 attempts to do this by:

   - creating model classes that represent search requests/responses
   - creating a search interface that accepts these model classes as input
   and return parameters
   - creating a controller that exposes this interface over REST
   - using Spring's IOC framework to select the correct implementation

An implementation of a search function was included in METRON-1022 as an
example.  ElasticsearchServiceImpl implements SearchService and is selected
as the implementation by default.  This could have been a separate PR but I
felt having it in context would help reviewers understand the design

How does this relate to METRON-990?  Currently they overlap with
METRON-1022 offering a subset of the functionality in the
METRON-990 Elasticsearch service.  The idea is to first ensure METRON-990
and METRON-1022 both conform to the same search abstraction (which has been
discussed in METRON-990 feedback).  The next step would be to replace the
search service in METRON-990 to one that queries the Metron REST service
instead.  Ideally this only involves changing one class since the
abstraction is used in all the other components of METRON-990 and is
trivial since the complexity is now in Metron REST and not javascript.
Next other services (getting index metadata for example) would be converted
using the same process in incremental PRs.  Then, moving forward, all
Elasticsearch interactions would instead be developed as Metron REST
endpoints using the foundation established in METRON-1022.

This is a lot to digest so I'm happy to more detail as needed.  Interested
to hear others' thoughts and reactions.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message