metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Merriman <merrim...@gmail.com>
Subject Re: [DISCUSS] Upcoming Release
Date Sat, 18 Nov 2017 02:28:05 GMT
Makes sense now.  Thanks Matt.

> On Nov 17, 2017, at 4:25 PM, Matt Foley <mfoley@hortonworks.com> wrote:
> 
> Hi Ryan,
> Yes and no.  The last release (see https://dist.apache.org/repos/dist/release/metron/
) was 0.4.1, announced on 9/19.
> Immediately after that we bumped the <version> of builds from master branch, per
https://issues.apache.org/jira/browse/METRON-1196 .  This is consistent with the Release Process
“Clean up” phase: “It is good practice to increment the build version in master immediately
after a Feature Release, so that dev builds with new stuff from master cannot be mistaken
for builds of the release version. So, immediately after a release, increment the MINOR version
number (eg, with the 0.4.0 just released, set the new version number to 0.4.1)” (https://cwiki.apache.org/confluence/display/METRON/Release+Process#ReleaseProcess-Step15-Cleanup
)
> 
> So you’re correct that the version of development builds is currently set to 0.4.2.
 After we actually make a release of 0.4.2, we’ll change dev build <version> to 0.4.3
(regardless of whether we expect the following release to be 0.4.3 or 0.5.0).
> 
> Hope this clarifies,
> --Matt
> 
> On 11/17/17, 1:59 PM, "Ryan Merriman" <merrimanr@gmail.com> wrote:
> 
>    Matt,
> 
>    I think we are currently on version 0.4.2.  If that is the case would the
>    next version be 0.4.3?
> 
>    Ryan
> 
>>    On Fri, Nov 17, 2017 at 3:31 PM, Matt Foley <mattf@apache.org> wrote:
>> 
>> (With release manager hat on)
>> 
>> The community has proposed a release of Metron in the near future,
>> focusing on Meta-alerts running in Elasticsearch.
>> Congrats on getting so many of the below already done.  At this point,
>> only METRON-1252, and the discussion of how to handle joint release of the
>> Metron bro plugin, remain as gating items for the release.  I project these
>> will be resolved next week, so let’s propose the following:
>> 
>> Sometime next week, after the last bits are done, I’ll start the release
>> process and create the release branch.
>> 
>> The proposed new version will be 0.4.2, unless there are backward
>> incompatible changes that support making it 0.5.0.
>> Currently there are NO included Jiras labeled ‘backward-incompatible’, nor
>> having Docs Text indicating so.
>> ***If anyone knows that some of the commits included since 0.4.1 introduce
>> backward incompatibility, please say so now on this thread, and mark the
>> Jira as such.***
>> 
>> The 90 or so jiras/commits already in master branch since 0.4.1 are listed
>> below.
>> Thanks,
>> --Matt
>> 
>>    METRON-1301 Alerts UI - Sorting on Triage Score Unexpectedly Filters
>> Some Records (nickwallen) closes apache/metron#832
>>    METRON-1294 IP addresses are not formatted correctly in facet and
>> group results (merrimanr) closes apache/metron#827
>>    METRON-1291 Kafka produce REST endpoint does not work in a Kerberized
>> cluster (merrimanr) closes apache/metron#826
>>    METRON-1290 Only first 10 alerts are update when a MetaAlert status is
>> changed to inactive (justinleet) closes apache/metron#842
>>    METRON-1311 Service Check Should Check Elasticsearch Index Templates
>> (nickwallen) closes apache/metron#839
>>    METRON-1289 Alert fields are lost when a MetaAlert is created
>> (merrimanr) closes apache/metron#824
>>    METRON-1309 Change metron-deployment to pull the plugin from
>> apache/metron-bro-plugin-kafka (JonZeolla) closes apache/metron#837
>>    METRON-1310 Template Delete Action Deletes Search Indices (nickwallen)
>> closes apache/metron#838
>>    METRON-1275: Fix Metron Documentation closes
>> apache/incubator-metron#833
>>    METRON-1295 Unable to Configure Logging for REST API (nickwallen)
>> closes apache/metron#828
>>    METRON-1307 Force install of java8 since java9 does not appear to work
>> with the scripts (brianhurley via ottobackwards) closes apache/metron#835
>>    METRON-1296 Full Dev Fails to Deploy Index Templates (nickwallen via
>> cestella) closes apache/incubator-metron#829
>>    METRON-1281 Remove hard-coded indices from the Alerts UI (merrimanr)
>> closes apache/metron#821
>>    METRON-1287 Full Dev Fails When Installing EPEL Repository
>> (nickwallen) closes apache/metron#820
>>    METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list
>> page (iraghumitra via merrimanr) closes apache/metron#819
>>    METRON-1283 Install Elasticsearch template as a part of the mpack
>> startup scripts (anandsubbu via nickwallen) closes apache/metron#817
>>    METRON-1254: Conditionals as map keys do not function in Stellar
>> closes apache/incubator-metron#801
>>    METRON-1261 Apply bro security patch (JonZeolla via ottobackwards)
>> closes apache/metron#805
>>    METRON-1284 Remove extraneous dead query in ElasticsearchDao
>> (justinleet) closes apache/metron#818
>>    METRON-1270: fix for warnings missing @return tag argument in
>> metron-analytics/metron-profiler-common and metron-profiler-client closes
>> apache/incubator-metron#810
>>    METRON-1272 Hide child alerts from searches and grouping if they
>> belong to meta alerts (justinleet) closes apache/metron#811
>>    METRON-1224 Add time range selection to search control (iraghumitra
>> via james-sirota) closes apache/metron#796
>>    METRON-1280 0.4.1 -> 0.4.2 missed a couple of projects (cestella via
>> justinleet) closes apache/metron#816
>>    METRON-1243: Add a REST endpoint which allows us to get a list of all
>> indice closes apache/incubator-metron#797
>>    METRON-1196 Increment master version number to 0.4.2 for on-going
>> development (mattf-horton) closes apache/metron#767
>>    METRON-1278 Strip &quot;Build Status&quot; widget from root README.md
>> in site-book build (mattf-horton) closes apache/metron#815
>>    METRON-1274 Master has failure in StormControllerIntegrationTest
>> (merrimanr) closes apache/metron#813
>>    METRON-1266 Profiler - SASL Authentication Failed (nickwallen) closes
>> apache/metron#809
>>    METRON-1260 Include Alerts UI in Ambari Service Check (nickwallen)
>> closes apache/metron#804
>>    METRON-1251: Typo and formatting fixes for metron-rest README closes
>> apache/incubator-metron#800
>>    METRON-1241: Enable the REST API to use a cache for the zookeeper
>> config similar to the Bolts closes apache/incubator-metron#795
>>    METRON-1267 Alerts UI returns a 404 when refreshing the alerts-list
>> page (merrimanr) closes apache/metron#808
>>    METRON-1262 Unable to add comment for a alert in a meta-alert
>> (merrimanr) closes apache/metron#806
>>    METRON-1263 Start Alerts UI service after Metron REST (anandsubbu via
>> nickwallen) closes apache/metron#807
>>    METRON-1255 MetaAlert search is not filtering on status (merrimanr)
>> closes apache/metron#802
>>    METRON-1249 Improve Metron MPack Service Checks (nickwallen) closes
>> apache/metron#799
>>    METRON-1237 address javadoc warnings in metron-maas-common (dbist via
>> james-sirota) closes apache/metron#792
>>    METRON-1240 address javadoc warnings in metron-platform and
>> metron-analytics (dbist via james-sirota) closes apache/metron#794
>>    METRON-1226 Searching Can Errantly Query the Wrong Indices
>> (nickwallen) closes apache/metron#793
>>    METRON-1081 Fix Alerts and Ops UI Notices file (james-sirota) closes
>> apache/metron#682
>>    METRON-1123 Add group by option using faceted search capabilities of
>> metron-rest-api (iraghumitra via james-sirota) closes apache/metron#768
>>    METRON-1223 Add support to add comments for alerts (iraghumitra via
>> james-sirota) closes apache/metron#788
>>    METRON-1083 Add filters using faceted search capabilities of
>> metron-rest-api (iraghumitra via james-sirota) closes apache/metron#710
>>    METRON-1232 Alert status changes are not reflected in list view
>> (iraghumitra via merrimanr) closes apache/metron#787
>>    METRON-1247 REST search and findOne endpoints return unexpected or
>> incorrect results for guids (justinleet) closes apache/metron#798
>>    METRON-1235: Document the properties pulled from the global
>> configuration closes apache/incubator-metron#791
>>    METRON-1234: fix for WARNING 'dependencies.dependency.(
>> groupId:artifactId:type:classifier)' must be unique:
>> org.apache.hadoop:hadoop-yarn-api:jar (dbist via mmiklavc) closes
>> apache/metron#790
>>    METRON-1222: fix warning for The expression ${parent.version} is
>> deprecated. Please use ${project.parent.version} instead. (dbist via
>> mmiklavc) closes apache/metron#782
>>    METRON-1220 Create documentation around alert nested field
>> (justinleet) closes apache/metron#780
>>    METRON-1229 Management UI type is part of the declarations of 2
>> modules (merrimanr) closes apache/metron#784
>>    METRON-1228: Configuration Management PUSH immediately does DUMP after
>> (mmiklavc via mmiklavc) closes apache/metron#783
>>    METRON-1218 Metron REST should return better error messages
>> (merrimanr) closes apache/metron#779
>>    METRON-1161 Add ability to edit parser command line options in the
>> management UI (merrimanr) closes apache/metron#737
>>    METRON-1209: Make stellar repl take logging properties, like other CLI
>> apps in metron closes apache/incubator-metron#772
>>    METRON-1059 address checkstyle warning AvoidStarImport in
>> metron-stellar (dbist via ottobackwards) closes apache/metron#664
>>    METRON-1204 UI does not time out after being idle, but stops
>> functioning (merrimanr) closes apache/metron#771
>>    METRON-1052: Add forensic similarity hash functions to Stellar closes
>> apache/incubator-metron#781
>>    METRON-632: Added validation of "shew.enrichmentType" and
>> "shew.keyColumns" closes apache/incubator-metron#732
>>    METRON-1194 Add Profiler Debug Functions to Profiler README
>> (nickwallen via ottobackwards) closes apache/metron#765
>>    METRON-1055 Metron 0.4.0 manual installation guide for CentOS 6
>> updates (lvets via ottobackwards) closes apache/metron#661
>>    METRON-1079 STELLAR NaN should be a keyword (ottobackwards) closes
>> apache/metron#681
>>    METRON-1085 Add REST endpoint to save a user profile for the Alerts UI
>> (merrimanr) closes apache/metron#694
>>    METRON-1208 MPack for Alerts UI (merrimanr) closes apache/metron#778
>>    METRON-1207 Make RPMs for Alerts UI (merrimanr) closes
>> apache/metron#777
>>    METRON-1215 Fix link to RPMs chapter (DimDroll via justinleet) closes
>> apache/metron#776
>>    METRON-1206 Make alerts UI conform to ops UI for install (merrimanr)
>> closes apache/metron#773
>>    METRON-1195 Meta alerts improperly handle updates to non-alert fields
>> (justinleet) closes apache/metron#766
>>    METRON-1189 Add alert escalation to the Alerts UI (merrimanr) closes
>> apache/metron#762
>>    METRON-1156 Simulate Triage Rules in the Stellar REPL (nickwallen)
>> closes apache/metron#733
>>    METRON-1198 Pycapa - No such configuration property
>> 'sasl.kerberos.principal' (nickwallen) closes apache/metron#769
>>    METRON-1202 ElasticsearchDao Has extraneous sleep call (justinleet)
>> closes apache/metron#770
>>    METRON-938 "service metron-rest start <password>" does not work on
>> CentOS 7. (justinleet) closes apache/metron#757
>>    METRON-1182 Refactor Code in alert list to accommodate new view types
>> (iraghumitra via merrimanr) closes apache/metron#756
>>    METRON-1188: Ambari global configuration management (mmiklavc) closes
>> apache/metron#760
>>    METRON-1191 update public web site to point at 0.4.1 new release
>> (mattf-horton) closes apache/metron#764
>>    METRON-1063 address javadoc warnings in metron-stellar (dbist via
>> ottobackwards) closes apache/metron#668
>>    METRON-1190 Fix Meta Alert Type handling in calculation of scores
>> (justinleet) closes apache/metron#763
>>    METRON-1187 Indexing/Profiler Kafka ACL Groups Not Setup Correctly
>> (nickwallen) closes apache/metron#759
>>    METRON-1185: Stellar REPL does not work on a kerberized cluster when
>> calling functions interacting with HBase closes apache/incubator-metron#755
>>    METRON-1186: Profiler Functions use classutils from shaded storm
>> closes apache/incubator-metron#758
>>    METRON-1173: Fix pointers to old stellar docs closes
>> apache/incubator-metron#746
>>    METRON-1179: Make STATS_ADD to take a list closes
>> apache/incubator-metron#750
>>    METRON-1180: Make Stellar Shell accept zookeeper quorum as a CSV list
>> and not require a port closes apache/incubator-metron#751
>>    METRON-1183 Improve KDC Setup Instructions (nickwallen) closes
>> apache/metron#753
>>    METRON-1177 Stale running topologies seen post-kerberization and cause
>> exceptions (nickwallen) closes apache/metron#748
>>    METRON-1158 Build backend for grouping alerts into meta alerts
>> (justinleet) closes apache/metron#734
>>    METRON-1146: Add ability to parse JSON string into JSONObject for
>> stellar closes apache/incubator-metron#727
>>    METRON-1176 REST: HDFS Service should support setting permissions on
>> files when writing (ottobackwards) closes apache/metron#749
>>    METRON-1114 Add group by capabilities to search REST endpoint
>> (merrimanr) closes apache/metron#702
>>    METRON-1167 Define Session Specific Global Configuration Values in the
>> REPL (nickwallen) closes apache/metron#740
>>    METRON-1171: Better validation for the SUBSTRING stellar function
>> closes apache/incubator-metron#745
>> 
>> 
>> 
>> On 11/17/17, 11:59 AM, "Nick Allen" <nick@nickallen.org> wrote:
>> 
>>    I just wanted to send an update on where we are at.  We've gotten a lot
>>    done here recently as you can see below.
>> 
>>      ✓ DONE (1) First, METRON-1289 needs to go in.  This one was a fairly
>> big
>>    effort and I am hearing that we are pretty close.
>> 
>>      ✓ DONE (2) METRON-1294 fixes an issue in how field types are
>> looked-up.
>> 
>>      ✓ DONE (3) METRON-1290 is next.  While this may have been fixed in
>>    M-1289, there may be some test cases we want from this PR.
>> 
>>      ✓ DONE (4) METRON-1301 addresses a problem with the sorting logic.
>> 
>>      ✓ DONE (5) METRON-1291 fixes an issue with escalation of metaalerts.
>> 
>>      (6) That leads us to Raghu's UI work in METRON-1252.  This
>> introduces the
>>    UI bits that depend on all the previous backend work.
>> 
>>      (7) At this point, we should have our best effort at running
>> Metaalerts
>>    on Elasticsearch 2.x. I propose that we cut a release here.
>> 
>>      (8) After we cut the release, we can introduce the work for ES 5.x in
>>    METRON-939.  I know we will need lots of help testing and reviewing
>> this
>>    one.
>> 
>> 
>> 
>>    We also have an outstanding question that needs resolved BEFORE we
>>    release.  We need to come to a consensus on how to release having
>> moved our
>>    Bro Plugin to a separate repo.  I don't think we've heard from
>> everyone on
>>    this.  I'd urge everyone to chime in so we can choose a path forward.
>> 
>>    If anyone is totally confused in regards to that discussion, I can try
>> and
>>    send an options summary again as a separate discuss thread.  The
>> original
>>    chain was somewhere around here [1].
>> 
>>    [1]
>>    https://lists.apache.org/thread.html/54a4474881b97e559df24728b3a0e9
>> 23a58345a282451085eef832ef@%3Cdev.metron.apache.org%3E
>> 
>> 
>> 
>>    On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <nick@nickallen.org>
>> wrote:
>> 
>>> Hi Guys -
>>> 
>>> I want to follow-up on this discussion.  It sounds like most people
>> are in
>>> agreement with the general approach.
>>> 
>>> A lot of people have been working hard on Metaalerts and
>> Elasticsearch.  I
>>> have checked-in with those doing the heavy lifting and have compiled
>> a more
>>> detailed plan based on where we are at now.  To the best of my
>> knowledge
>>> here is the plan of attack for finishing out this effort.
>>> 
>>>  (1) First, METRON-1289 needs to go in.  This one was a fairly big
>> effort
>>> and I am hearing that we are pretty close.
>>> 
>>>  (2) METRON-1294 fixes an issue in how field types are looked-up.
>>> 
>>>  (3) METRON-1290 is next.  While this may have been fixed in M-1289,
>>> there may be some test cases we want from this PR.
>>> 
>>>  (4) METRON-1301 addresses a problem with the sorting logic.
>>> 
>>>  (5) METRON-1291 fixes an issue with escalation of metaalerts.
>>> 
>>>  (6) That leads us to Raghu's UI work in METRON-1252.  This
>> introduces
>>> the UI bits that depend on all the previous backend work.
>>> 
>>>  (7) At this point, we should have our best effort at running
>> Metaalerts
>>> on Elasticsearch 2.x. I propose that we cut a release here.
>>> 
>>>  (8) After we cut the release, we can introduce the work for ES 5.x
>> in
>>> METRON-939.  I know we will need lots of help testing and reviewing
>> this
>>> one.
>>> 
>>> Please correct me if I am wrong.  I will try and send out updates as
>> we
>>> make progress.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <zeolla@gmail.com>
>> wrote:
>>> 
>>>> I agree, I think it's very reasonable to move in line with Nick's
>>>> proposal.  I would also suggest that we outline what the target
>> versions
>>>> would be to add in the METRON-777 components, since it has been
>> functional
>>>> for a very long time but not reviewed and has some really rockstar
>>>> improvements.
>>>> 
>>>> Jon
>>>> 
>>>> On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
>> ottobackwards@gmail.com>
>>>> wrote:
>>>> 
>>>>> I think the ES cutover should be the start of the 0.5.x series,
>> and we
>>>>> continue on with 0.4.x for the
>>>>> metadata improvements etc.  We could chose to focus 0.5.x’s first
>>>> releases
>>>>> on not only ES but
>>>>> getting a handle on kibana and the mpack situation as well.
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On November 6, 2017 at 12:48:45, Michael Miklavcic (
>>>>> michael.miklavcic@gmail.com) wrote:
>>>>> 
>>>>> I agree with your proposal, Nick. I think having a stabilizing
>> release
>>>>> prior to upgrading ES/Kibana makes sense.
>>>>> 
>>>>> On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen <nick@nickallen.org>
>> wrote:
>>>>> 
>>>>>> I would like to start a discussion around upcoming releases. We
>> have a
>>>>>> couple separate significant tracks of work that we need to
>> reconcile
>>>> in
>>>>> our
>>>>>> release schedule.
>>>>>> 
>>>>>> (1) We have had (and have in review) a good number of bug fixes
>>>> required
>>>>> to
>>>>>> support Metaalerts on the existing Elasticsearch 2.x
>> infrastructure.
>>>>>> 
>>>>>> 
>>>>>> (2) We also have ongoing work to upgrade our infrastructure to
>>>>>> Elasticsearch 5.x, which will not be backwards compatible.
>>>>>> 
>>>>>> 
>>>>>> I would like to see a release that has our best work on ES 2.x
>> before
>>>> we
>>>>>> migrate to 5.x. I would propose the following.
>>>>>> 
>>>>>> Release N+1: Introduce Metaalerts running on ES 2.x
>>>>>> 
>>>>>> Release N+2: Cut-over to ES 5.x
>>>>>> 
>>>>>> 
>>>>>> (Q) Is it worth cutting a separate release for ES 2.x? Is there
>> a
>>>> better
>>>>>> way to handle the cut-over to 5.x?
>>>>>> 
>>>>> 
>>>> --
>>>> 
>>>> Jon
>>>> 
>>> 
>>> 
>> 
>> 
>> 
>> 
> 
> 

Mime
View raw message