metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Merriman <>
Subject [DISCUSS] Stellar REST client
Date Fri, 19 Oct 2018 15:46:22 GMT
I want to open up discussion around adding a Stellar REST client function.
There are services available to enrich security telemetry and they are
commonly exposed through a REST interface.  The primary purpose of this
discuss thread to collect requirements from the community and agree on a
general architectural approach.

At a minimum I see a Stellar REST client supporting:

   - Common HTTP verbs including GET, POST, DELETE, etc
   - Option to provide headers and request parameters as needed
   - Support for basic authentication
   - Proper request and error handling (we can discuss further how this
   should work)
   - SSL support
   - Option to use a proxy server (including authentication)
   - JSON format

In addition to these functional requirements, I would also propose we
include these performance requirements:

   - Provide a configurable caching layer
   - Provide a mechanism for pooling connections
   - Provide clear documentation and guidance on how to properly use this
   feature since there is a significant risk of introducing latency issues

What else would you like to see included?

I think the primary architectural decision we need to make (based on the
agreed upon requirements of course) is an appropriate Java HTTP/REST client
library.  Ideally we choose a library that supports everything we need
OOTB.  I think the majority of the work for this feature will involve
wrapping this library in a Stellar function and exposing the configuration
knobs through Metron's configuration interface (Ambari, Zookeeper, etc).  I
have done some very light research and here is my initial list:

   - Apache HttpComponents -
   - Has support for all of the features listed above as far as I can tell
      - Doesn't introduce a large number of new dependencies (am I wrong
      - Is sort of included already (we will need to upgrade from
      - Lower level
   - Google HTTP Client Library for Java -
   - Higher level API with pluggable components
      - Introduces dependencies (we've had issues with Guava in the past)
   - Netflix Ribbon -
      - Has a lot of nice features that may be useful in the future
      - Introduces dependencies (including guava)
      - Hasn't been committed to in the last 5-6 months
   - Unirest -
      - Lightweight API built on top of HttpComponents
      - Pluggable serialization library (jackson is an issue for us so this
      is nice)
      - Also has not received a commit in a while
   - OkHttp -
   - Good documentation and looks easy to use
      - Actively maintained

Obviously we have a lot of choices.  I think it comes down to balancing the
tradeoff between ease of use (HttpComponents will likely require the most
work since it is lower level) and capability.  Introducing additional
dependencies is something we should also be mindful of because our shading

This should get us started.  Let me know what you think!

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message