metron-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mmiklavc <...@git.apache.org>
Subject [GitHub] metron pull request #1242: METRON-1834: Migrate Elasticsearch from Transport...
Date Fri, 19 Oct 2018 20:53:47 GMT
GitHub user mmiklavc opened a pull request:

    https://github.com/apache/metron/pull/1242

    METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API

    ## Contributor Comments
    
    https://issues.apache.org/jira/browse/METRON-1834
    
    This task has been a long time coming after having completed the ES upgrade in https://issues.apache.org/jira/browse/METRON-939.
Motivation for completing this now is that Elasticsearch will be deprecating use of the TransportClient
in v 7.x. This PR migrates the Elasticsearch client from TransportClient to the newer Java
REST API. 
    
    1. https://www.elastic.co/guide/en/elasticsearch/client/java-api/5.6/client.html
    2. https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/java-rest-overview.html
    3. https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/java-rest-high-level-migration.html
    
    This builds off and finishes work started by @cestella here - https://github.com/cestella/incubator-metron/tree/es_rest_client.
I condensed his branch into 1 flattened commit and built on top of it in order to provide
attribution.
    
    I have a number of tasks I'm still working through, but I wanted to get the review process
started. I've minimally validated X-Pack auth and will have some follow-up for SSL. Test plans
and a breakdown of the changes will be soon to follow. For starters, full dev should continue
to work as normal and you should see data flowing into indexes for bro, snort, and yaf. There
are some additional changes to how this client will be configured, which I'll be documenting
shortly. The new client does not take a Map of settings any longer now that it is leverage
Apache HTTP Async Client https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/java-rest-low-usage-dependencies.html
under the hood. This meant choosing a set of properties to expose and doing a translation
to the builder pattern under the hood. Again, I'll have a write-up of this in the migration
guide and update the README's accordingly.
    
    NOTE: This checks off 2 items from this follow-on list https://github.com/apache/metron/pull/840#issuecomment-347281776
    
    1. Fix Log4j logging problem - classpath issues
    2. Migrate to new ES REST client
    
    Per discussion in the Metron Slack channel, I will be updating the Jira ticket with a
series of tasks to be completed prior to acceptance, including performance regression testing
compared with the old API.
    
    ## Pull Request Checklist
    
    Thank you for submitting a contribution to Apache Metron.  
    Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.  
    Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.  
    
    
    In order to streamline the review of the contribution we ask you follow these guidelines
and ask you to double check the following:
    
    ### For all changes:
    - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at
[Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
    - [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are
trying to resolve? Pay particular attention to the hyphen "-" character.
    - [x] Has your PR been rebased against the latest commit within the target branch (typically
master)?
    
    
    ### For code changes:
    - [ ] Have you included steps to reproduce the behavior or problem that is being changed
or addressed?
    - [ ] Have you included steps or a guide to how the change may be verified and tested
manually?
    - [ ] Have you ensured that the full suite of tests and checks have been executed in the
root metron folder via:
      ```
      mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh

      ```
    
    - [ ] Have you written or updated unit tests and or integration tests to verify your changes?
    - [ ] If adding new dependencies to the code, are these dependencies licensed in a way
that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
    - [ ] Have you verified the basic functionality of the build by building and running locally
with Vagrant full-dev environment or the equivalent?
    
    ### For documentation related changes:
    - [ ] Have you ensured that format looks appropriate for the output in which it is rendered
by building and verifying the site-book? If not then run the following commands and the verify
changes via `site-book/target/site/index.html`:
    
      ```
      cd site-book
      mvn site
      ```
    
    #### Note:
    Please ensure that once the PR is submitted, you check travis-ci for build issues and
submit an update to your PR as soon as possible.
    It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal
repository such that your branches are built there before submitting a pull request.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/mmiklavc/metron es-rest-client

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/metron/pull/1242.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1242
    
----
commit a7c7dc287b4f9c99c6780b934a0b6f433a03aa04
Author: cstella <cestella@...>
Date:   2018-10-09T00:06:52Z

    Casey Stella - elasticsearch rest client migration base work

commit 10410ea9718a2a1b1d287fb4f22a6c98efb1fdaa
Author: Michael Miklavcic <michael.miklavcic@...>
Date:   2018-10-09T00:07:22Z

    Update shade plugin version

commit a33a16872118175ed35729df6ddde2959e49ae2f
Author: Michael Miklavcic <michael.miklavcic@...>
Date:   2018-10-09T15:56:08Z

    Fix es update dao test

commit 52c3c96d7657205e65d3bd0c0e35923a851911da
Author: Michael Miklavcic <michael.miklavcic@...>
Date:   2018-10-09T21:26:16Z

    Merge with master. Fix es search integration tests

commit 4742832869f9512e11cab4a32109c15a2b17a92e
Author: Michael Miklavcic <michael.miklavcic@...>
Date:   2018-10-11T18:45:06Z

    Merge branch 'master' into es-rest-client

commit 43809968320e586fd70411776140f3aa13a60195
Author: Michael Miklavcic <michael.miklavcic@...>
Date:   2018-10-11T23:59:25Z

    Get shade plugin working with the new ES client and the ClassIndexTransformer Shade plugin
transformer.

commit af03f6f036e96c742db39733bf8ebc2cbf229129
Author: Michael Miklavcic <michael.miklavcic@...>
Date:   2018-10-19T02:56:52Z

    Introduce config classes for managing ES client configuration. Translate properties for
new client.

commit dfcf9df2c9d41ddd777d02678c90fbb1e7dce1a3
Author: Michael Miklavcic <michael.miklavcic@...>
Date:   2018-10-19T19:21:23Z

    Resolve merge conflicts with master

commit babe86451abe0f56a2cd5ed92e49cbed6bca31ae
Author: Michael Miklavcic <michael.miklavcic@...>
Date:   2018-10-19T19:50:31Z

    Merge branch 'master' into es-rest-client

commit df2a77d15251d7c1ea77f87d238a0fba932ae4dd
Author: Michael Miklavcic <michael.miklavcic@...>
Date:   2018-10-19T20:19:50Z

    Remove extra deps in elasticsearch pom

----


---

Mime
View raw message