metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zeolla@GMail.com" <zeo...@gmail.com>
Subject Re: [DISCUSS] Upgrading HBase and Kafka support
Date Fri, 08 Mar 2019 15:30:13 GMT
+1 to option 3 on both.  Also strongly in favor of Docker.  We recently
took a similar approach in metron-bro-plugin-kafka as well (link
<https://github.com/apache/metron-bro-plugin-kafka/tree/master/docker>) to
do end to end testing.

Jon

On Fri, Mar 8, 2019 at 9:53 AM Nick Allen <nick@nickallen.org> wrote:

> +1 for option 3.  I am in favor of using Docker for the integration tests
> for all the reasons that you mentioned.
>
> On Fri, Mar 8, 2019 at 9:47 AM Ryan Merriman <merrimanr@gmail.com> wrote:
>
> > I have been researching the effort involved to upgrade to HDP 3.  Along
> the
> > way I've found a couple challenging issues that we will need to solve,
> both
> > involving our integration testing strategy.
> >
> > The first issue is Kafka.  We are moving from 0.10.0 to 2.0.0 and there
> > have been significant changes to the API.  This creates an issue in the
> > KafkaComponent class, which we use as an in-memory Kafka server in
> > integration tests.  Most of the classes that were previously used have
> gone
> > away, and to the best of my knowledge, were not supported as public APIs.
> > I also don't see any publicly documented APIs to replace them.
> >
> > The second issue is HBase.  We are moving from 1.1.2 to 2.0.2 so another
> > significant change.  This creates an issue in the MockHTable class
> > becausethe HTableInterface class has changed to Table, essentially
> > requiring that MockHTable be rewritten to conform to the new interface.
> > It's my opinion that this class is complicated and difficult to maintain
> as
> > it is anyways.
> >
> > These 2 issues have the potential to add a significant amount of work to
> > upgrading Metron to HDP 3.  I want to take a step back and review our
> > options before we move forward.  Here are some initial thoughts I had on
> > how to approach this.  For HBase:
> >
> >    1. Update MockHTable to work with the new HBase API.  We would
> continue
> >    using a mock server approach for HBase.
> >    2. Research replacing MockHTable with an in-memory HBase server.
> >    3. Replace MockHTable with a Docker container running HBase.
> >
> > For Kafka:
> >
> >    1. Replace KafkaComponent with a mock server implementation.
> >    2. Update KafkaComponent to work with the new API.  We would probably
> >    need to leverage some internal Kafka classes.  I do not see a testing
> > API
> >    documented publicly.
> >    3. Replace KafkaComponent with a Docker container running Kafka.
> >
> > What other options are there?  Whatever we choose I think we should
> follow
> > a similar approach for both (mock servers, in memory servers, Docker,
> other
> > options I'm not thinking of).
> >
> > This will not shock anyone but I would be in favor of Docker containers.
> > They have the advantage of classpath isolation, easy upgrades, and
> accurate
> > integration testing.  The downside is we will have to adjusts our tests
> and
> > travis script to incorporate these Docker containers into our build
> > process.  We have discussed this at length in the past and it has
> generally
> > stalled for various reasons.  Maybe if we move a few services at a time
> it
> > might be more palatable?  As for the other 2 approaches, I think if
> either
> > worked well we wouldn't be having this discussion.  Mock servers are hard
> > to maintain and I don't see in memory testing classes documented in
> > javadocs for either service.
> >
> > Thoughts?
> >
>
-- 

Jon Zeolla

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message