metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Foley <ma...@apache.org>
Subject Re: [DISCUSS] Upcoming Release
Date Mon, 04 Dec 2017 21:57:32 GMT
Hey all, 
I see METRON-1252 was resolved over the weekend.  Shall I go ahead and start the process with
0.4.2 release?
Does anyone have any commits they feel strongly should go in before 0.4.2 is done, or are
we ready to call it good?

I believe there is consensus the 0.4.2 release should include a release of the current state
of the metron-bro-plugin-kafka.  I will continue the discussion in that thread as to the process
for accomplishing that, but plan on it happening.

Regards,
--Matt

On 11/26/17, 6:26 PM, "Matt Foley" <mattf@apache.org> wrote:

    Hope everyone (at least in the U.S.) had a great Thanksgiving holiday.
    Regarding status of the release effort, still pending METRON-1252, so not making the release
branch yet.
    
    Regards,
    --Matt
    
    On 11/17/17, 1:32 PM, "Matt Foley" <mattf@apache.org> wrote:
    
        (With release manager hat on)
        
        The community has proposed a release of Metron in the near future, focusing on Meta-alerts
running in Elasticsearch.
        Congrats on getting so many of the below already done.  At this point, only METRON-1252,
and the discussion of how to handle joint release of the Metron bro plugin, remain as gating
items for the release.  I project these will be resolved next week, so let’s propose the
following:
        
        Sometime next week, after the last bits are done, I’ll start the release process
and create the release branch.
        
        The proposed new version will be 0.4.2, unless there are backward incompatible changes
that support making it 0.5.0.
        Currently there are NO included Jiras labeled ‘backward-incompatible’, nor having
Docs Text indicating so.
        ***If anyone knows that some of the commits included since 0.4.1 introduce backward
incompatibility, please say so now on this thread, and mark the Jira as such.***
        
        The 90 or so jiras/commits already in master branch since 0.4.1 are listed below.
        Thanks,
        --Matt
        
            METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly Filters Some Records
(nickwallen) closes apache/metron#832
            METRON-1294 IP addresses are not formatted correctly in facet and group results
(merrimanr) closes apache/metron#827
            METRON-1291 Kafka produce REST endpoint does not work in a Kerberized cluster
(merrimanr) closes apache/metron#826
            METRON-1290 Only first 10 alerts are update when a MetaAlert status is changed
to inactive (justinleet) closes apache/metron#842
            METRON-1311 Service Check Should Check Elasticsearch Index Templates (nickwallen)
closes apache/metron#839
            METRON-1289 Alert fields are lost when a MetaAlert is created (merrimanr) closes
apache/metron#824
            METRON-1309 Change metron-deployment to pull the plugin from apache/metron-bro-plugin-kafka
(JonZeolla) closes apache/metron#837
            METRON-1310 Template Delete Action Deletes Search Indices (nickwallen) closes
apache/metron#838
            METRON-1275: Fix Metron Documentation closes apache/incubator-metron#833
            METRON-1295 Unable to Configure Logging for REST API (nickwallen) closes apache/metron#828
            METRON-1307 Force install of java8 since java9 does not appear to work with the
scripts (brianhurley via ottobackwards) closes apache/metron#835
            METRON-1296 Full Dev Fails to Deploy Index Templates (nickwallen via cestella)
closes apache/incubator-metron#829
            METRON-1281 Remove hard-coded indices from the Alerts UI (merrimanr) closes apache/metron#821
            METRON-1287 Full Dev Fails When Installing EPEL Repository (nickwallen) closes
apache/metron#820
            METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list page (iraghumitra
via merrimanr) closes apache/metron#819
            METRON-1283 Install Elasticsearch template as a part of the mpack startup scripts
(anandsubbu via nickwallen) closes apache/metron#817
            METRON-1254: Conditionals as map keys do not function in Stellar closes apache/incubator-metron#801
            METRON-1261 Apply bro security patch (JonZeolla via ottobackwards) closes apache/metron#805
            METRON-1284 Remove extraneous dead query in ElasticsearchDao (justinleet) closes
apache/metron#818
            METRON-1270: fix for warnings missing @return tag argument in metron-analytics/metron-profiler-common
and metron-profiler-client closes apache/incubator-metron#810
            METRON-1272 Hide child alerts from searches and grouping if they belong to meta
alerts (justinleet) closes apache/metron#811
            METRON-1224 Add time range selection to search control (iraghumitra via james-sirota)
closes apache/metron#796
            METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects (cestella via justinleet)
closes apache/metron#816
            METRON-1243: Add a REST endpoint which allows us to get a list of all indice closes
apache/incubator-metron#797
            METRON-1196 Increment master version number to 0.4.2 for on-going development
(mattf-horton) closes apache/metron#767
            METRON-1278 Strip &quot;Build Status&quot; widget from root README.md
in site-book build (mattf-horton) closes apache/metron#815
            METRON-1274 Master has failure in StormControllerIntegrationTest (merrimanr) closes
apache/metron#813
            METRON-1266 Profiler - SASL Authentication Failed (nickwallen) closes apache/metron#809
            METRON-1260 Include Alerts UI in Ambari Service Check (nickwallen) closes apache/metron#804
            METRON-1251: Typo and formatting fixes for metron-rest README closes apache/incubator-metron#800
            METRON-1241: Enable the REST API to use a cache for the zookeeper config similar
to the Bolts closes apache/incubator-metron#795
            METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list page (merrimanr)
closes apache/metron#808
            METRON-1262 Unable to add comment for a alert in a meta-alert (merrimanr) closes
apache/metron#806
            METRON-1263 Start Alerts UI service after Metron REST (anandsubbu via nickwallen)
closes apache/metron#807
            METRON-1255 MetaAlert search is not filtering on status (merrimanr) closes apache/metron#802
            METRON-1249 Improve Metron MPack Service Checks (nickwallen) closes apache/metron#799
            METRON-1237 address javadoc warnings in metron-maas-common (dbist via james-sirota)
closes apache/metron#792
            METRON-1240 address javadoc warnings in metron-platform and metron-analytics (dbist
via james-sirota) closes apache/metron#794
            METRON-1226 Searching Can Errantly Query the Wrong Indices (nickwallen) closes
apache/metron#793
            METRON-1081 Fix Alerts and Ops UI Notices file (james-sirota) closes apache/metron#682
            METRON-1123 Add group by option using faceted search capabilities of metron-rest-api
(iraghumitra via james-sirota) closes apache/metron#768
            METRON-1223 Add support to add comments for alerts (iraghumitra via james-sirota)
closes apache/metron#788
            METRON-1083 Add filters using faceted search capabilities of metron-rest-api (iraghumitra
via james-sirota) closes apache/metron#710
            METRON-1232 Alert status changes are not reflected in list view (iraghumitra via
merrimanr) closes apache/metron#787
            METRON-1247 REST search and findOne endpoints return unexpected or incorrect results
for guids (justinleet) closes apache/metron#798
            METRON-1235: Document the properties pulled from the global configuration closes
apache/incubator-metron#791
            METRON-1234: fix for WARNING 'dependencies.dependency.(groupId:artifactId:type:classifier)'
must be unique: org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc) closes apache/metron#790
            METRON-1222: fix warning for The expression ${parent.version} is deprecated. Please
use ${project.parent.version} instead. (dbist via mmiklavc) closes apache/metron#782
            METRON-1220 Create documentation around alert nested field (justinleet) closes
apache/metron#780
            METRON-1229 Management UI type is part of the declarations of 2 modules (merrimanr)
closes apache/metron#784
            METRON-1228: Configuration Management PUSH immediately does DUMP after (mmiklavc
via mmiklavc) closes apache/metron#783
            METRON-1218 Metron REST should return better error messages (merrimanr) closes
apache/metron#779
            METRON-1161 Add ability to edit parser command line options in the management
UI (merrimanr) closes apache/metron#737
            METRON-1209: Make stellar repl take logging properties, like other CLI apps in
metron closes apache/incubator-metron#772
            METRON-1059 address checkstyle warning AvoidStarImport in metron-stellar (dbist
via ottobackwards) closes apache/metron#664
            METRON-1204 UI does not time out after being idle, but stops functioning (merrimanr)
closes apache/metron#771
            METRON-1052: Add forensic similarity hash functions to Stellar closes apache/incubator-metron#781
            METRON-632: Added validation of "shew.enrichmentType" and "shew.keyColumns" closes
apache/incubator-metron#732
            METRON-1194 Add Profiler Debug Functions to Profiler README (nickwallen via ottobackwards)
closes apache/metron#765
            METRON-1055 Metron 0.4.0 manual installation guide for CentOS 6 updates (lvets
via ottobackwards) closes apache/metron#661
            METRON-1079 STELLAR NaN should be a keyword (ottobackwards) closes apache/metron#681
            METRON-1085 Add REST endpoint to save a user profile for the Alerts UI (merrimanr)
closes apache/metron#694
            METRON-1208 MPack for Alerts UI (merrimanr) closes apache/metron#778
            METRON-1207 Make RPMs for Alerts UI (merrimanr) closes apache/metron#777
            METRON-1215 Fix link to RPMs chapter (DimDroll via justinleet) closes apache/metron#776
            METRON-1206 Make alerts UI conform to ops UI for install (merrimanr) closes apache/metron#773
            METRON-1195 Meta alerts improperly handle updates to non-alert fields (justinleet)
closes apache/metron#766
            METRON-1189 Add alert escalation to the Alerts UI (merrimanr) closes apache/metron#762
            METRON-1156 Simulate Triage Rules in the Stellar REPL (nickwallen) closes apache/metron#733
            METRON-1198 Pycapa - No such configuration property 'sasl.kerberos.principal'
(nickwallen) closes apache/metron#769
            METRON-1202 ElasticsearchDao Has extraneous sleep call (justinleet) closes apache/metron#770
            METRON-938 "service metron-rest start <password>" does not work on CentOS
7. (justinleet) closes apache/metron#757
            METRON-1182 Refactor Code in alert list to accommodate new view types (iraghumitra
via merrimanr) closes apache/metron#756
            METRON-1188: Ambari global configuration management (mmiklavc) closes apache/metron#760
            METRON-1191 update public web site to point at 0.4.1 new release (mattf-horton)
closes apache/metron#764
            METRON-1063 address javadoc warnings in metron-stellar (dbist via ottobackwards)
closes apache/metron#668
            METRON-1190 Fix Meta Alert Type handling in calculation of scores (justinleet)
closes apache/metron#763
            METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup Correctly (nickwallen)
closes apache/metron#759
            METRON-1185: Stellar REPL does not work on a kerberized cluster when calling functions
interacting with HBase closes apache/incubator-metron#755
            METRON-1186: Profiler Functions use classutils from shaded storm closes apache/incubator-metron#758
            METRON-1173: Fix pointers to old stellar docs closes apache/incubator-metron#746
            METRON-1179: Make STATS_ADD to take a list closes apache/incubator-metron#750
            METRON-1180: Make Stellar Shell accept zookeeper quorum as a CSV list and not
require a port closes apache/incubator-metron#751
            METRON-1183 Improve KDC Setup Instructions (nickwallen) closes apache/metron#753
            METRON-1177 Stale running topologies seen post-kerberization and cause exceptions
(nickwallen) closes apache/metron#748
            METRON-1158 Build backend for grouping alerts into meta alerts (justinleet) closes
apache/metron#734
            METRON-1146: Add ability to parse JSON string into JSONObject for stellar closes
apache/incubator-metron#727
            METRON-1176 REST: HDFS Service should support setting permissions on files when
writing (ottobackwards) closes apache/metron#749
            METRON-1114 Add group by capabilities to search REST endpoint (merrimanr) closes
apache/metron#702
            METRON-1167 Define Session Specific Global Configuration Values in the REPL (nickwallen)
closes apache/metron#740
            METRON-1171: Better validation for the SUBSTRING stellar function closes apache/incubator-metron#745
        
        
        
        On 11/17/17, 11:59 AM, "Nick Allen" <nick@nickallen.org> wrote:
        
            I just wanted to send an update on where we are at.  We've gotten a lot
            done here recently as you can see below.
            
              ✓ DONE (1) First, METRON-1289 needs to go in.  This one was a fairly big
            effort and I am hearing that we are pretty close.
            
              ✓ DONE (2) METRON-1294 fixes an issue in how field types are looked-up.
            
              ✓ DONE (3) METRON-1290 is next.  While this may have been fixed in
            M-1289, there may be some test cases we want from this PR.
            
              ✓ DONE (4) METRON-1301 addresses a problem with the sorting logic.
            
              ✓ DONE (5) METRON-1291 fixes an issue with escalation of metaalerts.
            
              (6) That leads us to Raghu's UI work in METRON-1252.  This introduces the
            UI bits that depend on all the previous backend work.
            
              (7) At this point, we should have our best effort at running Metaalerts
            on Elasticsearch 2.x. I propose that we cut a release here.
            
              (8) After we cut the release, we can introduce the work for ES 5.x in
            METRON-939.  I know we will need lots of help testing and reviewing this
            one.
            
            
            
            We also have an outstanding question that needs resolved BEFORE we
            release.  We need to come to a consensus on how to release having moved our
            Bro Plugin to a separate repo.  I don't think we've heard from everyone on
            this.  I'd urge everyone to chime in so we can choose a path forward.
            
            If anyone is totally confused in regards to that discussion, I can try and
            send an options summary again as a separate discuss thread.  The original
            chain was somewhere around here [1].
            
            [1]
            https://lists.apache.org/thread.html/54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@%3Cdev.metron.apache.org%3E
            
            
            
            On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <nick@nickallen.org> wrote:
            
            > Hi Guys -
            >
            > I want to follow-up on this discussion.  It sounds like most people are in
            > agreement with the general approach.
            >
            > A lot of people have been working hard on Metaalerts and Elasticsearch. 
I
            > have checked-in with those doing the heavy lifting and have compiled a more
            > detailed plan based on where we are at now.  To the best of my knowledge
            > here is the plan of attack for finishing out this effort.
            >
            >   (1) First, METRON-1289 needs to go in.  This one was a fairly big effort
            > and I am hearing that we are pretty close.
            >
            >   (2) METRON-1294 fixes an issue in how field types are looked-up.
            >
            >   (3) METRON-1290 is next.  While this may have been fixed in M-1289,
            > there may be some test cases we want from this PR.
            >
            >   (4) METRON-1301 addresses a problem with the sorting logic.
            >
            >   (5) METRON-1291 fixes an issue with escalation of metaalerts.
            >
            >   (6) That leads us to Raghu's UI work in METRON-1252.  This introduces
            > the UI bits that depend on all the previous backend work.
            >
            >   (7) At this point, we should have our best effort at running Metaalerts
            > on Elasticsearch 2.x. I propose that we cut a release here.
            >
            >   (8) After we cut the release, we can introduce the work for ES 5.x in
            > METRON-939.  I know we will need lots of help testing and reviewing this
            > one.
            >
            > Please correct me if I am wrong.  I will try and send out updates as we
            > make progress.
            >
            >
            >
            >
            >
            > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <zeolla@gmail.com>
wrote:
            >
            >> I agree, I think it's very reasonable to move in line with Nick's
            >> proposal.  I would also suggest that we outline what the target versions
            >> would be to add in the METRON-777 components, since it has been functional
            >> for a very long time but not reviewed and has some really rockstar
            >> improvements.
            >>
            >> Jon
            >>
            >> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <ottobackwards@gmail.com>
            >> wrote:
            >>
            >> > I think the ES cutover should be the start of the 0.5.x series,
and we
            >> > continue on with 0.4.x for the
            >> > metadata improvements etc.  We could chose to focus 0.5.x’s first
            >> releases
            >> > on not only ES but
            >> > getting a handle on kibana and the mpack situation as well.
            >> >
            >> >
            >> >
            >> >
            >> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
            >> > michael.miklavcic@gmail.com) wrote:
            >> >
            >> > I agree with your proposal, Nick. I think having a stabilizing release
            >> > prior to upgrading ES/Kibana makes sense.
            >> >
            >> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <nick@nickallen.org>
wrote:
            >> >
            >> > > I would like to start a discussion around upcoming releases.
We have a
            >> > > couple separate significant tracks of work that we need to
reconcile
            >> in
            >> > our
            >> > > release schedule.
            >> > >
            >> > > (1) We have had (and have in review) a good number of bug fixes
            >> required
            >> > to
            >> > > support Metaalerts on the existing Elasticsearch 2.x infrastructure.
            >> > >
            >> > >
            >> > > (2) We also have ongoing work to upgrade our infrastructure
to
            >> > > Elasticsearch 5.x, which will not be backwards compatible.
            >> > >
            >> > >
            >> > > I would like to see a release that has our best work on ES
2.x before
            >> we
            >> > > migrate to 5.x. I would propose the following.
            >> > >
            >> > > Release N+1: Introduce Metaalerts running on ES 2.x
            >> > >
            >> > > Release N+2: Cut-over to ES 5.x
            >> > >
            >> > >
            >> > > (Q) Is it worth cutting a separate release for ES 2.x? Is there
a
            >> better
            >> > > way to handle the cut-over to 5.x?
            >> > >
            >> >
            >> --
            >>
            >> Jon
            >>
            >
            >
            
        
        
        
    



Mime
View raw message