metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Casey Stella <ceste...@gmail.com>
Subject Re: [DISCUSS] Upcoming Release
Date Mon, 04 Dec 2017 22:01:31 GMT
I would be in favor of a release at this point.

On Mon, Dec 4, 2017 at 4:57 PM, Matt Foley <mattf@apache.org> wrote:

> Hey all,
> I see METRON-1252 was resolved over the weekend.  Shall I go ahead and
> start the process with 0.4.2 release?
> Does anyone have any commits they feel strongly should go in before 0.4.2
> is done, or are we ready to call it good?
>
> I believe there is consensus the 0.4.2 release should include a release of
> the current state of the metron-bro-plugin-kafka.  I will continue the
> discussion in that thread as to the process for accomplishing that, but
> plan on it happening.
>
> Regards,
> --Matt
>
> On 11/26/17, 6:26 PM, "Matt Foley" <mattf@apache.org> wrote:
>
>     Hope everyone (at least in the U.S.) had a great Thanksgiving holiday.
>     Regarding status of the release effort, still pending METRON-1252, so
> not making the release branch yet.
>
>     Regards,
>     --Matt
>
>     On 11/17/17, 1:32 PM, "Matt Foley" <mattf@apache.org> wrote:
>
>         (With release manager hat on)
>
>         The community has proposed a release of Metron in the near future,
> focusing on Meta-alerts running in Elasticsearch.
>         Congrats on getting so many of the below already done.  At this
> point, only METRON-1252, and the discussion of how to handle joint release
> of the Metron bro plugin, remain as gating items for the release.  I
> project these will be resolved next week, so let’s propose the following:
>
>         Sometime next week, after the last bits are done, I’ll start the
> release process and create the release branch.
>
>         The proposed new version will be 0.4.2, unless there are backward
> incompatible changes that support making it 0.5.0.
>         Currently there are NO included Jiras labeled
> ‘backward-incompatible’, nor having Docs Text indicating so.
>         ***If anyone knows that some of the commits included since 0.4.1
> introduce backward incompatibility, please say so now on this thread, and
> mark the Jira as such.***
>
>         The 90 or so jiras/commits already in master branch since 0.4.1
> are listed below.
>         Thanks,
>         --Matt
>
>             METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly
> Filters Some Records (nickwallen) closes apache/metron#832
>             METRON-1294 IP addresses are not formatted correctly in facet
> and group results (merrimanr) closes apache/metron#827
>             METRON-1291 Kafka produce REST endpoint does not work in a
> Kerberized cluster (merrimanr) closes apache/metron#826
>             METRON-1290 Only first 10 alerts are update when a MetaAlert
> status is changed to inactive (justinleet) closes apache/metron#842
>             METRON-1311 Service Check Should Check Elasticsearch Index
> Templates (nickwallen) closes apache/metron#839
>             METRON-1289 Alert fields are lost when a MetaAlert is created
> (merrimanr) closes apache/metron#824
>             METRON-1309 Change metron-deployment to pull the plugin from
> apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
>             METRON-1310 Template Delete Action Deletes Search Indices
> (nickwallen) closes apache/metron#838
>             METRON-1275: Fix Metron Documentation closes
> apache/incubator-metron#833
>             METRON-1295 Unable to Configure Logging for REST API
> (nickwallen) closes apache/metron#828
>             METRON-1307 Force install of java8 since java9 does not appear
> to work with the scripts (brianhurley via ottobackwards) closes
> apache/metron#835
>             METRON-1296 Full Dev Fails to Deploy Index Templates
> (nickwallen via cestella) closes apache/incubator-metron#829
>             METRON-1281 Remove hard-coded indices from the Alerts UI
> (merrimanr) closes apache/metron#821
>             METRON-1287 Full Dev Fails When Installing EPEL Repository
> (nickwallen) closes apache/metron#820
>             METRON-1267 Alerts UI returns a 404 when refreshing the
> alerts-list page (iraghumitra via merrimanr) closes apache/metron#819
>             METRON-1283 Install Elasticsearch template as a part of the
> mpack startup scripts (anandsubbu via nickwallen) closes apache/metron#817
>             METRON-1254: Conditionals as map keys do not function in
> Stellar closes apache/incubator-metron#801
>             METRON-1261 Apply bro security patch (JonZeolla via
> ottobackwards) closes apache/metron#805
>             METRON-1284 Remove extraneous dead query in ElasticsearchDao
> (justinleet) closes apache/metron#818
>             METRON-1270: fix for warnings missing @return tag argument in
> metron-analytics/metron-profiler-common and metron-profiler-client closes
> apache/incubator-metron#810
>             METRON-1272 Hide child alerts from searches and grouping if
> they belong to meta alerts (justinleet) closes apache/metron#811
>             METRON-1224 Add time range selection to search control
> (iraghumitra via james-sirota) closes apache/metron#796
>             METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects
> (cestella via justinleet) closes apache/metron#816
>             METRON-1243: Add a REST endpoint which allows us to get a list
> of all indice closes apache/incubator-metron#797
>             METRON-1196 Increment master version number to 0.4.2 for
> on-going development (mattf-horton) closes apache/metron#767
>             METRON-1278 Strip &quot;Build Status&quot; widget from root
> README.md in site-book build (mattf-horton) closes apache/metron#815
>             METRON-1274 Master has failure in
> StormControllerIntegrationTest (merrimanr) closes apache/metron#813
>             METRON-1266 Profiler - SASL Authentication Failed (nickwallen)
> closes apache/metron#809
>             METRON-1260 Include Alerts UI in Ambari Service Check
> (nickwallen) closes apache/metron#804
>             METRON-1251: Typo and formatting fixes for metron-rest README
> closes apache/incubator-metron#800
>             METRON-1241: Enable the REST API to use a cache for the
> zookeeper config similar to the Bolts closes apache/incubator-metron#795
>             METRON-1267 Alerts UI returns a 404 when refreshing the
> alerts-list page (merrimanr) closes apache/metron#808
>             METRON-1262 Unable to add comment for a alert in a meta-alert
> (merrimanr) closes apache/metron#806
>             METRON-1263 Start Alerts UI service after Metron REST
> (anandsubbu via nickwallen) closes apache/metron#807
>             METRON-1255 MetaAlert search is not filtering on status
> (merrimanr) closes apache/metron#802
>             METRON-1249 Improve Metron MPack Service Checks (nickwallen)
> closes apache/metron#799
>             METRON-1237 address javadoc warnings in metron-maas-common
> (dbist via james-sirota) closes apache/metron#792
>             METRON-1240 address javadoc warnings in metron-platform and
> metron-analytics (dbist via james-sirota) closes apache/metron#794
>             METRON-1226 Searching Can Errantly Query the Wrong Indices
> (nickwallen) closes apache/metron#793
>             METRON-1081 Fix Alerts and Ops UI Notices file (james-sirota)
> closes apache/metron#682
>             METRON-1123 Add group by option using faceted search
> capabilities of metron-rest-api (iraghumitra via james-sirota) closes
> apache/metron#768
>             METRON-1223 Add support to add comments for alerts
> (iraghumitra via james-sirota) closes apache/metron#788
>             METRON-1083 Add filters using faceted search capabilities of
> metron-rest-api (iraghumitra via james-sirota) closes apache/metron#710
>             METRON-1232 Alert status changes are not reflected in list
> view (iraghumitra via merrimanr) closes apache/metron#787
>             METRON-1247 REST search and findOne endpoints return
> unexpected or incorrect results for guids (justinleet) closes
> apache/metron#798
>             METRON-1235: Document the properties pulled from the global
> configuration closes apache/incubator-metron#791
>             METRON-1234: fix for WARNING 'dependencies.dependency.(
> groupId:artifactId:type:classifier)' must be unique:
> org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc) closes
> apache/metron#790
>             METRON-1222: fix warning for The expression ${parent.version}
> is deprecated. Please use ${project.parent.version} instead. (dbist via
> mmiklavc) closes apache/metron#782
>             METRON-1220 Create documentation around alert nested field
> (justinleet) closes apache/metron#780
>             METRON-1229 Management UI type is part of the declarations of
> 2 modules (merrimanr) closes apache/metron#784
>             METRON-1228: Configuration Management PUSH immediately does
> DUMP after (mmiklavc via mmiklavc) closes apache/metron#783
>             METRON-1218 Metron REST should return better error messages
> (merrimanr) closes apache/metron#779
>             METRON-1161 Add ability to edit parser command line options in
> the management UI (merrimanr) closes apache/metron#737
>             METRON-1209: Make stellar repl take logging properties, like
> other CLI apps in metron closes apache/incubator-metron#772
>             METRON-1059 address checkstyle warning AvoidStarImport in
> metron-stellar (dbist via ottobackwards) closes apache/metron#664
>             METRON-1204 UI does not time out after being idle, but stops
> functioning (merrimanr) closes apache/metron#771
>             METRON-1052: Add forensic similarity hash functions to Stellar
> closes apache/incubator-metron#781
>             METRON-632: Added validation of "shew.enrichmentType" and
> "shew.keyColumns" closes apache/incubator-metron#732
>             METRON-1194 Add Profiler Debug Functions to Profiler README
> (nickwallen via ottobackwards) closes apache/metron#765
>             METRON-1055 Metron 0.4.0 manual installation guide for CentOS
> 6 updates (lvets via ottobackwards) closes apache/metron#661
>             METRON-1079 STELLAR NaN should be a keyword (ottobackwards)
> closes apache/metron#681
>             METRON-1085 Add REST endpoint to save a user profile for the
> Alerts UI (merrimanr) closes apache/metron#694
>             METRON-1208 MPack for Alerts UI (merrimanr) closes
> apache/metron#778
>             METRON-1207 Make RPMs for Alerts UI (merrimanr) closes
> apache/metron#777
>             METRON-1215 Fix link to RPMs chapter (DimDroll via justinleet)
> closes apache/metron#776
>             METRON-1206 Make alerts UI conform to ops UI for install
> (merrimanr) closes apache/metron#773
>             METRON-1195 Meta alerts improperly handle updates to non-alert
> fields (justinleet) closes apache/metron#766
>             METRON-1189 Add alert escalation to the Alerts UI (merrimanr)
> closes apache/metron#762
>             METRON-1156 Simulate Triage Rules in the Stellar REPL
> (nickwallen) closes apache/metron#733
>             METRON-1198 Pycapa - No such configuration property
> 'sasl.kerberos.principal' (nickwallen) closes apache/metron#769
>             METRON-1202 ElasticsearchDao Has extraneous sleep call
> (justinleet) closes apache/metron#770
>             METRON-938 "service metron-rest start <password>" does not
> work on CentOS 7. (justinleet) closes apache/metron#757
>             METRON-1182 Refactor Code in alert list to accommodate new
> view types (iraghumitra via merrimanr) closes apache/metron#756
>             METRON-1188: Ambari global configuration management (mmiklavc)
> closes apache/metron#760
>             METRON-1191 update public web site to point at 0.4.1 new
> release (mattf-horton) closes apache/metron#764
>             METRON-1063 address javadoc warnings in metron-stellar (dbist
> via ottobackwards) closes apache/metron#668
>             METRON-1190 Fix Meta Alert Type handling in calculation of
> scores (justinleet) closes apache/metron#763
>             METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup
> Correctly (nickwallen) closes apache/metron#759
>             METRON-1185: Stellar REPL does not work on a kerberized
> cluster when calling functions interacting with HBase closes
> apache/incubator-metron#755
>             METRON-1186: Profiler Functions use classutils from shaded
> storm closes apache/incubator-metron#758
>             METRON-1173: Fix pointers to old stellar docs closes
> apache/incubator-metron#746
>             METRON-1179: Make STATS_ADD to take a list closes
> apache/incubator-metron#750
>             METRON-1180: Make Stellar Shell accept zookeeper quorum as a
> CSV list and not require a port closes apache/incubator-metron#751
>             METRON-1183 Improve KDC Setup Instructions (nickwallen) closes
> apache/metron#753
>             METRON-1177 Stale running topologies seen post-kerberization
> and cause exceptions (nickwallen) closes apache/metron#748
>             METRON-1158 Build backend for grouping alerts into meta alerts
> (justinleet) closes apache/metron#734
>             METRON-1146: Add ability to parse JSON string into JSONObject
> for stellar closes apache/incubator-metron#727
>             METRON-1176 REST: HDFS Service should support setting
> permissions on files when writing (ottobackwards) closes apache/metron#749
>             METRON-1114 Add group by capabilities to search REST endpoint
> (merrimanr) closes apache/metron#702
>             METRON-1167 Define Session Specific Global Configuration
> Values in the REPL (nickwallen) closes apache/metron#740
>             METRON-1171: Better validation for the SUBSTRING stellar
> function closes apache/incubator-metron#745
>
>
>
>         On 11/17/17, 11:59 AM, "Nick Allen" <nick@nickallen.org> wrote:
>
>             I just wanted to send an update on where we are at.  We've
> gotten a lot
>             done here recently as you can see below.
>
>               ✓ DONE (1) First, METRON-1289 needs to go in.  This one was
> a fairly big
>             effort and I am hearing that we are pretty close.
>
>               ✓ DONE (2) METRON-1294 fixes an issue in how field types are
> looked-up.
>
>               ✓ DONE (3) METRON-1290 is next.  While this may have been
> fixed in
>             M-1289, there may be some test cases we want from this PR.
>
>               ✓ DONE (4) METRON-1301 addresses a problem with the sorting
> logic.
>
>               ✓ DONE (5) METRON-1291 fixes an issue with escalation of
> metaalerts.
>
>               (6) That leads us to Raghu's UI work in METRON-1252.  This
> introduces the
>             UI bits that depend on all the previous backend work.
>
>               (7) At this point, we should have our best effort at running
> Metaalerts
>             on Elasticsearch 2.x. I propose that we cut a release here.
>
>               (8) After we cut the release, we can introduce the work for
> ES 5.x in
>             METRON-939.  I know we will need lots of help testing and
> reviewing this
>             one.
>
>
>
>             We also have an outstanding question that needs resolved
> BEFORE we
>             release.  We need to come to a consensus on how to release
> having moved our
>             Bro Plugin to a separate repo.  I don't think we've heard from
> everyone on
>             this.  I'd urge everyone to chime in so we can choose a path
> forward.
>
>             If anyone is totally confused in regards to that discussion, I
> can try and
>             send an options summary again as a separate discuss thread.
> The original
>             chain was somewhere around here [1].
>
>             [1]
>             https://lists.apache.org/thread.html/
> 54a4474881b97e559df24728b3a0e923a58345a282451085eef832ef@%
> 3Cdev.metron.apache.org%3E
>
>
>
>             On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <
> nick@nickallen.org> wrote:
>
>             > Hi Guys -
>             >
>             > I want to follow-up on this discussion.  It sounds like most
> people are in
>             > agreement with the general approach.
>             >
>             > A lot of people have been working hard on Metaalerts and
> Elasticsearch.  I
>             > have checked-in with those doing the heavy lifting and have
> compiled a more
>             > detailed plan based on where we are at now.  To the best of
> my knowledge
>             > here is the plan of attack for finishing out this effort.
>             >
>             >   (1) First, METRON-1289 needs to go in.  This one was a
> fairly big effort
>             > and I am hearing that we are pretty close.
>             >
>             >   (2) METRON-1294 fixes an issue in how field types are
> looked-up.
>             >
>             >   (3) METRON-1290 is next.  While this may have been fixed
> in M-1289,
>             > there may be some test cases we want from this PR.
>             >
>             >   (4) METRON-1301 addresses a problem with the sorting logic.
>             >
>             >   (5) METRON-1291 fixes an issue with escalation of
> metaalerts.
>             >
>             >   (6) That leads us to Raghu's UI work in METRON-1252.  This
> introduces
>             > the UI bits that depend on all the previous backend work.
>             >
>             >   (7) At this point, we should have our best effort at
> running Metaalerts
>             > on Elasticsearch 2.x. I propose that we cut a release here.
>             >
>             >   (8) After we cut the release, we can introduce the work
> for ES 5.x in
>             > METRON-939.  I know we will need lots of help testing and
> reviewing this
>             > one.
>             >
>             > Please correct me if I am wrong.  I will try and send out
> updates as we
>             > make progress.
>             >
>             >
>             >
>             >
>             >
>             > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <
> zeolla@gmail.com> wrote:
>             >
>             >> I agree, I think it's very reasonable to move in line with
> Nick's
>             >> proposal.  I would also suggest that we outline what the
> target versions
>             >> would be to add in the METRON-777 components, since it has
> been functional
>             >> for a very long time but not reviewed and has some really
> rockstar
>             >> improvements.
>             >>
>             >> Jon
>             >>
>             >> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> ottobackwards@gmail.com>
>             >> wrote:
>             >>
>             >> > I think the ES cutover should be the start of the 0.5.x
> series, and we
>             >> > continue on with 0.4.x for the
>             >> > metadata improvements etc.  We could chose to focus
> 0.5.x’s first
>             >> releases
>             >> > on not only ES but
>             >> > getting a handle on kibana and the mpack situation as
> well.
>             >> >
>             >> >
>             >> >
>             >> >
>             >> > On November 6, 2017 at 12:48:45, Michael Miklavcic (
>             >> > michael.miklavcic@gmail.com) wrote:
>             >> >
>             >> > I agree with your proposal, Nick. I think having a
> stabilizing release
>             >> > prior to upgrading ES/Kibana makes sense.
>             >> >
>             >> > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <
> nick@nickallen.org> wrote:
>             >> >
>             >> > > I would like to start a discussion around upcoming
> releases. We have a
>             >> > > couple separate significant tracks of work that we need
> to reconcile
>             >> in
>             >> > our
>             >> > > release schedule.
>             >> > >
>             >> > > (1) We have had (and have in review) a good number of
> bug fixes
>             >> required
>             >> > to
>             >> > > support Metaalerts on the existing Elasticsearch 2.x
> infrastructure.
>             >> > >
>             >> > >
>             >> > > (2) We also have ongoing work to upgrade our
> infrastructure to
>             >> > > Elasticsearch 5.x, which will not be backwards
> compatible.
>             >> > >
>             >> > >
>             >> > > I would like to see a release that has our best work on
> ES 2.x before
>             >> we
>             >> > > migrate to 5.x. I would propose the following.
>             >> > >
>             >> > > Release N+1: Introduce Metaalerts running on ES 2.x
>             >> > >
>             >> > > Release N+2: Cut-over to ES 5.x
>             >> > >
>             >> > >
>             >> > > (Q) Is it worth cutting a separate release for ES 2.x?
> Is there a
>             >> better
>             >> > > way to handle the cut-over to 5.x?
>             >> > >
>             >> >
>             >> --
>             >>
>             >> Jon
>             >>
>             >
>             >
>
>
>
>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message